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Color Image Processing Apparatus and Pattern 
Extracting Apparatus 

Background of the Invention 
5 Field of the Invention 

The present invention relates to a color image 
processing apparatus and a pattern extracting 
apparatus, in particular, to those suitable for 
extracting a title or the like from a color image. 

10 

Description of the Related Art 

In recent years, as computers and their peripheral 
units such as printers have become common and their 
costs have been decreased, color images have been used 

15 in a variety of fields. Thus, a technology for 
dividing a color image into several areas and 
extracting only a particular area has been desired. 
For example, a technology for extracting areas with 
the same color from a color image has been desired. 

20 When a color scenery image photographed by a CCD 
camera is used for an input image to be process, such 
a technology has been widely desired in many 
applications for selecting fruit and watching cars and 
people for securities. 

25 When a color document image is used as an input 



image, such a technology is expected to automatically 
extract a document name and a keyword from the image. 
Examples of such a technology are data retrieval 
systems such as book categorizing systems in libraries 
and automatic management systems. In addition, such 
a technology is used for automatically assigning 
keywords and file names in groupware for storing and 
sharing image data as a database. Such information 
can be used for retrieving desired data from a large 
number of color document images. 

As technologies for extracting a particular area 
from a color image, the following methods have been 
proposed . 

(1) RGB color space clustering method 

In the RGB color space clustering method, a color 
separated image is generated. In other words, pixels 
of an RGB image photographed by a CCD camera are 
clustered in the RGB space. With pixels in one 
cluster, an image of each color is generated. Thus, 
areas with the same color are extracted. Areas 
generated in such a method are combined so as to 
extract a new area. 

Fig. 1 is a schematic diagram for explaining the 
conventional RGB color space clustering method. 

In Fig. 1, assuming that a color document image 
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501 is input , patterns with similar colors are 
extracted and clustered. For example, assuming that 
patterns 502 and 507 are blue group colors, a pattern 
503 is a green group color, and patterns 504 to 506 
5 are red group colors, a cluster 508 that includes the 
blue group color patterns 502 and 507, a cluster 509 
that includes the green group color pattern 503, and 
a cluster 510 that includes the red group color 
patterns 504 to 506 are generated in the RGB space. 

10 

When the clusters 508 to 510 are generated, images 
of the clusters 508 to 510 are generated with pixels 
that are included therein. Thus, for the cluster 508, 
a color separated image 501a composed of patterns 502 f 
15 and 507 1 is generated. For the cluster 509, a color 
separated image 501b composed of a pattern 503 1 is 
generated. For the cluster 510, a color separated 
image 501c composed of patterns 504' to 506 f is 
generated . 

20 ( 2 ) Non-RGB color space clustering method 

All pixels of an RGB image represented in the RGB 
space are converted into another color space such as 
HSV. The pixels are clustered in the color space in 
a particular manner. Images are generated with pixels 

25 that are included in each cluster so as to extract 
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areas with the same colors. The obtained areas are 
combined so as to extract a new area. 

The following technical papers that describe 
technologies for extracting character areas from color 
5 document images are known. 

Senda et. al., "Method for extracting a 
character pattern from a color image due to a single 
color of characters (translated title)", The Institute 
of Electronics, Information and Communication 
10 Engineers, Japan, PRU 94-04, pp 17-24, 

Uehane et. al., "Extracting a character area 
from a color image using iso-color line process 
(translated title)", The Institute of Electronics, 
Information and Communication Engineers, Japan, PRU 
15 94-09, pp 9-16, 

Matsuo et. al., "Extracting a single color 
character area from a color document image (translated 
title)", 1997 Annual Convention, The Institute of 
Electronics, Information and Communication Engineers, 
20 Japan, D-12-19, 

- Matsuo et. al., "Extracting a character string 
from a scenery image with gradation and color 
information (translated title)", The Institute of 
Electronics, Information and Communication Engineers, 
25 Japan, PRU 92-121, pp 25-32. 
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(3) Area expanding method 

In the area expanding method, adjacent pixels are 
assigned labels corresponding to only similarities of 
colors. In other words, the maximum value (max) of 
5 each color element of (R, G, B) of pixels represented 
with RGB is obtained and normalized as (R/max, G/max, 
B/max). Thus, a normalized image is generated. 

Fig. 2 is a schematic diagram showing the 
conventional area expanding method. 
10 In Fig. 2, pixels PI, P2, and so forth in an image 

are represented with color elements RGB as P1(R1, Gl, 
Bl), P2(R2, G2, B2), and so forth [1]. 

Next, the maximum value of each color element is 
obtained. For example, the maximum value of Rl, Gl, 
15 and Bl of a pixel PI is denoted by maxl . Likewise, 
the maximum value of R2, G2, and B2 of a pixel P2 is 
denoted by max2. With the maximum values, each color 
element is normalized. Thus, normalized pixels 
Pl f (Rl/maxl, Gl/maxl, Bl/maxl) and P2 f (R2/max2, 
20 G2/max2, B2/max2) are obtained [2]. 

The square of the difference of each color element 
of the normalized pixels PI' and P2 1 is obtained. The 
results are cumulated so as to obtain the distance 
between the adjacent pixels PI 1 and P2 1 as d = 
25 (Rl/maxl - R2/max2) 2 + (Gl/maxl - G2/max2) 2 + (Bl/maxl 
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- B2/max2) 2 [3] . 

Thus, when the distance d is smaller than a 
predetermined fixed threshold value THO, the pixels 
PI and P2 are treated as those with the same color and 
5 assigned the same label . After all the image is 
assigned labels, the same color areas with the same 
labels are extracted. 

In the area expanding method, since only adjacent 
pixels are processed, the process time of this method 
10 is shorter than that of the RGB color space clustering 
method . 

For details of the area expanding method, refer to 
Japanese Patent Laid-Open Publication No, 5-298443. 
In addition, as a method for extracting a 
15 character area from a color separated image, the 
above-mentioned method (by Uehane et. al, "Extracting 
a character area from a color image using iso-color 
line process", The Institute of Electronics, 
Information and Communication Engineers, Japan, PRU 
20 94-09, pp 9-16) is known. In this method, a character 
area is extracted from a color image with the 
following steps. 

- Enclosing rectangles in connected areas are 
obtained from a single-color separated image. 
25 - Enclosing rectangles are limited in a 
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predetermined range of the sizes and shapes thereof. 

An adjacent rectangle search range of each 
rectangle is assigned. Rectangles are searched as a 
group in each search range. 
5 - Rectangles with a high linearity of center of 

gravity are kept in each group. 

An enclosing rectangle of each group is 
obtained and a pattern with a color similar to a color 
of the area that composed the group is extracted. 

10 However, the conventional clustering method for 

clustering the same color area of a color image has 
the following problems. 

In the RGB color space clustering method or 
another color space clustering method (for example, 

15 HSV space clustering method), all pixels of the image 
are clustered. Thus, even if the color of the pattern 
502 is different from the color of the pattern 507, 
when their colors are similar to each other, the 
patterns 502 and 507 may be categorized as the same 

20 cluster 508. In this case, due to the color of the 
pattern 507, the shape of the cluster of the color of 
the pattern 502 is varied. Thus, the color range of 
the cluster of the color of the pattern 502 is 
distorted. Consequently, the pattern 502 cannot be 

25 accurately extracted. For example, when the pattern 
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502 to be extracted is apart from the pattern 507 with 
a similar color thereto, if they are extracted as one 
cluster 508, the color of the cluster 508 becomes a 
mixed color of the color of the pattern 502 and the 
5 color of the pattern 507. The color of the cluster 
508 cannot cover the color range of the pattern 502 
and the color range of the pattern 508. Thus, holes 
511 and 512 may take place in the patterns 502' and 
507 1 as the extracted results. Alternatively, 

10 contours of the patterns 502 1 and 507 1 may not be 
clearly extracted. 

In the RGB color space clustering method or 
another color space clustering method (for example, 
HSV color space clustering method), since many pixels 

15 of all the image are clustered, the calculating time 
for the clustering process becomes long. 

On the other hand, in the area expanding method, 
to normalize pixels as shown in Fig. 2, since 
divisions should be performed for each pixel, the 

20 number of calculations becomes large. The results of 
the divisions should be stored as floating-point data 
for all the pixels. Thus, the memory resource 
necessary for the process becomes large. Moreover, 
after the pixels are normalized, adjacent pixels that 

25 are equally viewed by the observer may largely deviate 
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from a fixed threshold value depending on the 
definition of similarities of colors of these pixels. 
Thus, holes may take place in an area. Alternatively, 
the contour of an area may not be correctly extracted. 
5 In addition, since only the relation of adjacent 
pixels is considered, when the color gradually varies 
at the boundary of the character area and the 
background area, the character area and the background 
area are assigned the same label. 

10 In the conventional character area extracting 

method, color separated images are generated 
corresponding to the number of colors of the entire 
image. Thus, it takes a long time to generate the 
color separated images. For example, when a title is 

15 extracted from the image, the title tends to be 
adversely affected by other colors. Thus, the 
extracting accuracy of the title deteriorates. When 
an enclosing rectangle of connected areas is obtained, 
the entire image should be processed for each of the 

20 extracted color separated images. Consequently, a 
plurality of images with the same size (corresponding 
to the number of extracted colors ) are required for 
a color image. Thus, it takes a long time to process 
the color image. 

25 In addition, since enclosing rectangles are 
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grouped corresponding to the individual color 
separated images of the entire color image, it takes 
a long time to process the color image. When 
characters to be extracted are clustered to different 
5 color separated images, they cannot be properly 
extracted . 

When patterns are grouped, only rectangles in 
relevant search ranges are extracted. Thus, small 
portions tend to be dropped from relevant groups. To 
10 restore dropped portions, patterns with similar colors 
are extracted at last. However, at this point, noise 
with a similar color tends to be extracted. 

Summary of the Invention 
15 A first object of the present invention is to 

provide a color image processing apparatus that 
performs a distance calculating process for a part of 
an image area without need to perform a normalizing 
process . 

20 A second object of the present invention is to 

provide a pattern extracting apparatus that extracts 
a pattern at high speed while suppressing the pattern 
from being dropped. 

A third object of the present invention is to 

25 provide a pattern extracting apparatus that allows a 



11 

pattern to be accurately extracted even if the 
boundary thereof is not clear. 

To solve the above described problem, an aspect of 
the present invention is a color image processing 
5 apparatus having a color image inputting unit for 
inputting a color image and outputting a color image 
signal, a holding unit for holing the color image 
signal, and a calculating unit for processing the 
color image signal, comprising a threshold value 

10 assigning unit for assigning a threshold value 
corresponding to color information of a considered 
pixel, and a labeling unit for comparing color 
information of adjacent pixels and assigning the same 
label to the pixels when the distance thereof is the 

15 threshold value or less. 

Thus, the threshold value of the distance for 
determining whether or not to assign the same label 
to the adjacent pixels can be assigned corresponding 
to color information of the considered pixel, not a 

20 fixed value. Thus, it can be accurately determined 
whether or not the pixels are in the same range. 
Consequently, pixels can be accurately extracted from 
a desired area. 

An aspect of the present invention is a color 

25 image processing apparatus further comprising an 
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average color calculating unit for obtaining the 
average color of pixels assigned the same label, and 
an average color comparing unit for comparing the 
color of a pixel adjacent to the considered pixel and 
5 the average color, wherein the labeling unit assigns 
the same label to a pixel only when the compared 
result is in a predetermined range. 

Thus, even if the color of an area to be extracted 
gradually varies and the boundary thereof is not 
10 clear, the area to be extracted and an area of a 
background thereof can be distinguished. 
Consequently, the extracting accuracy of the area can 
be improved . 

An aspect of the present invention is a color 
15 image processing apparatus having a color image 
inputting unit for inputting a color image and 
outputting a color image signal, a holding unit for 
holing the color image signal, and a calculating unit 
for processing the color image signal, comprising an 
20 image reducing unit for reducing the color image 
signal and generating a reduced image, wherein areas 
with the same color are obtained from the reduced 
image and areas corresponding to the obtained areas 
are extracted corresponding to only a color extracted 
25 from the reduced image. 
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Thus, areas regarded as the same color can be 
obtained with a reduced image. Consequently, a 
process for extracting areas regarded as the same 
color can be performed at high speed. 

An aspect of the present invention is a color 
image processing apparatus, wherein the labeling unit 
obtains enclosing rectangles of label images and label 
areas of the reduced image, representative colors 
thereof, the number of colors in each label area, and 
colors from the reduced image, obtains rectangles of 
the original image corresponding thereto, searches a 
pixel with a color similar to the representative color 
in the rectangle, and performs a labeling process 
starting from the searched pixel. 

Thus, enclosing rectangles of label images and 
labels, representative colors thereof, the number of 
colors in the same label area, and colors are obtained 
from a reduced image. Corresponding to the obtained 
results, a rectangle of the original image is 
obtained. The labeling process can be performed in 
the rectangle. Thus, the labeling process can be 
performed at high speed. 

An aspect of the present invention is a color 
image processing apparatus, wherein the labeling unit 
determines all colors and similarities of areas of the 
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reduced image and assigns the same label to areas when 
the similarities of the colors of the areas are 
smaller than the threshold value. 

Thus, all colors and their similarities of 
5 obtained from a reduced image and similarities of 
areas of the reduced image are determined. When the 
similarity of colors of the areas is smaller than the 
predetermined threshold value, the same label can be 
assigned to the areas. Thus, a desired area can be 

10 extracted as viewed by the naked eye. 

An aspect of the present invention is a color 
image processing apparatus, wherein the threshold 
value assigning unit determines the similarities of 
colors of areas with a variable threshold value of 

15 three color elements, determines that the similarities 
of colors of the areas are 0 when the color difference 
distance of the three color elements of the areas is 
smaller than the variable threshold value, and 
determines that the similarities of colors of the 

20 areas are larger than the threshold value when the 
color difference distance of the three color elements 
of the areas is equal to or larger than the threshold 
value . 

Thus, threshold value elements are assigned three 
25 color elements, respectively. When all the three 
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color elements of two colors are less than the 
respective threshold value (elements), these colors 
can be treated as the same color. Consequently, the 
capacity of the table of the threshold values can be 
decreased. In addition, the labeling process can be 
very accurately performed. 

An aspect of the present invention is a pattern 
extracting unit, comprising a categorizing unit for 
categorizing a pattern of an input image corresponding 
to color information and geometry information of the 
pattern of the input image. 

Thus, when a headline area with the same color is 
extracted from a color input image, a part of the 
color input image is limited as the headline area. 
Patterns are categorized corresponding to color 
information. A pattern apart from the headline area 
is removed from an object to be categorized. In 
addition, a small pattern and a large pattern that are 
not headline characters can be removed from an object 
to be categorized corresponding to color information. 
Thus, the speed of the extracting process of the 
headline area can be improved. 

An aspect of the present invention is a pattern 
extracting apparatus, comprising a clustering unit for 
clustering pixels of an input image corresponding to 
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color information of adjacent pixels, and a grouping 
unit for categorizing the clusters as groups 
corresponding to color information and geometry 
information of the clusters obtained by the clustering 
unit. 

Thus, to cluster pixels of an input image, color 
information of a considered pixel and color 
information of a pixel adjacent thereto are compared. 
When all pixels of the input image are compared, the 
number of times of the comparing process of color 
information of adjacent pixels becomes the square of 
the number of pixels of the input image. In contrast, 
when the color information of a considered pixel is 
compared with the color information of a pixel 
adjacent thereto, the number of times of the comparing 
process becomes the number of pixels of the input 
image. Thus, the clustering process can be performed 
at high speed. 

Since clusters are categorized as groups, pixels 
in the same cluster can be integrally handled. Thus, 
it is not necessary to process individual pixels. 
Consequently, the grouping process can be performed 
at high speed. In addition, in a limited range of the 
input area, clusters with similar colors can be 
categorized as the same group. Thus, the grouping 
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process can be performed at further higher speed. 

An aspect of the present invention is a pattern 
extracting apparatus for extracting areas in the range 
of the variation of colors determined in a second 
5 resolution as areas with the same color in the case 
that the areas are determined as the same color in a 
first resolution and as different colors in the second 
resolution. 

Thus, since the color of a pattern of an input 

10 image is represented in a combination of sizes of dots 
of basic colors, even if an area represented as the 
same color in the input image is detected as a set of 
different colors depending on a read resolution, when 
the input image is read, the range of the same color 

15 of the area can be accurately extracted. 

Consequently, a part of a pattern represented as the 
same color in the input image can be prevented from 
being dropped or lost. 

An aspect of the present invention is a pattern 

20 extracting apparatus, comprising a unit for generating 
an enclosing rectangle of a label image, a unit for 
extracting an enclosing rectangle with a size in a 
predetermined range from the extracted enclosing 
rectangle, a unit for assigning a search area in a 

25 predetermined area of the extracted enclosing 
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rectangle, and a unit for assigning an enclosing 
rectangle in the search area, or partially disposed 
in the search area as an alternative to be categorized 
as a group with respect to the extracted enclosing 
5 rectangle. 

Thus, among enclosing rectangles generated with 
label images, an enclosing rectangle that becomes a 
reference for assigning a search area can be limited 
to an enclosing rectangle with a predetermined size. 

10 Thus, when a character area is extracted from an input 
image, enclosing rectangles of a small pattern such 
as noise and a large pattern such as a background or 
a frame can be prevented from being selected as a 
reference of a search area for searching a character 

15 string. Consequently, a pattern that is not a 
character string can be suppressed from being 
categorized as a group. Thus, only a character string 
can be effectively extracted. 

Since a search area for searching a character 

20 string is assigned in a predetermined range of a 
considered enclosing rectangle, only characters in the 
vicinity of a considered enclosing rectangle can be 
searched. When a title or the like is extracted from 
the input image, a character that is apart from the 

25 title and that does not compose the title can be 
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prevented from being extracted as the group of the 
title. Thus, only characters of the title can be 
effectively extracted. 

An aspect of the present invention is a pattern 
5 extracting apparatus, comprising a first color 
information comparing unit for comparing color 
information of adjacent pixels of an input image, a 
second color information comparing unit for comparing 
the color information of the pixels compared by the 

10 first color information comparing unit and color 
information of a label image adjacent thereto, and a 
labeling unit for assigning labels to pixels 
corresponding to the compared result of the first 
color information comparing unit and the compared 

15 result of the second color information comparing unit. 

Thus, in the case that a color gradually varies, 
only when color information of adjacent pixels is 
compared, since the colors of the adjacent pixels are 
similar, these pixels are assigned the same label. 

20 When these pixels are traced, the difference of colors 
cumulates. Thus, pixels with different colors may be 
assigned the same label. However, according to the 
present invention, since color information of a 
pattern that has been assigned a label can be 

25 compared, the labeling process can be performed 
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corresponding to the cumulated value of the difference 
of colors. Thus, when an area whose color gradually 
varies is assigned a label corresponding to the 
compared result of the color information of the 
5 adjacent pixels, pixels with different colors can be 
prevented from being assigned the same label. 

An aspect of the present invention is a pattern 
extracting apparatus, comprising a color difference 
calculating unit for calculating the color difference 

10 between adjacent pixels of an area represented with 
a particular color, a threshold value assigning unit 
for assigning a threshold value corresponding to the 
color difference, and a labeling unit for assigning 
a label to a pixel adjacent to a pixel represented 

15 with the predetermined color corresponding to the 
threshold value. 

Thus, since an input image is printed by halftone 
printing method, even if the input image is 
represented in the same color, it may be detected as 

20 a set of different colors depending on a read 
resolution. In this case, an area with the same color 
can be accurately extracted. Thus, a part of a 
pattern can be prevented from being dropped or lost 
from an area represented as the same color in the 

25 input image. 
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An aspect of the present invention is a pattern 
extracting apparatus further comprising a print model 
generating unit having meshes of basic colors, basic 
color dots being disposed at mesh points of the 
5 meshes, the meshes being rotated and superimposed each 
other, the sizes of the basic color dots being varied, 
so as to generate a color as a print model . 

Thus, a color generated by the halftone printing 
method is virtually generated on a computer. The 
10 variation of a color by the halftone printing method 
can be distinguished without need to analyze the 
variation of the color by the halftone printing method 
using a real printed document. 

An aspect of the present invention is a pattern 
15 extracting apparatus further comprising a color 
difference table generating unit for generating a 
color difference table that stores luminance values 
of colors generated by the print model generating unit 
and the color difference between adjacent pixels 
20 corresponding to the luminance values in each read 
resolution of the print model. 

Thus, regardless of a resolution of a device that 
reads data from a real printed document, a threshold 
value for determining whether or not colors that are 
25 read from a real printed document are the same can be 
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easily obtained. 

An aspect: of "the present invention is a pattern 
extracting apparatus further comprising a model 
resolution estimation processing unit for estimating 
5 the read resolution of the print mode for an input 
image . 

Thus, a real printed document that is read by a 
device can be handled with a print model virtually 
generated on a computer. The labeling process can be 

10 performed with a threshold value generated with a 
print model to the real printed document that is read 
by the device. 

An aspect of the present invention is a pattern 
extracting apparatus, wherein the model resolution 

15 estimating processing unit estimates a resolution of 
which a luminance value of a color and a color 
difference registered in the color difference table 
optimally confirms to the entire input image as a read 
resolution of the print model of the input image. 

20 Thus, for a real printed document that is read by 

a device, an optimum threshold value generated by a 
print model is applied to the entire input image so 
as to perform the labeling process. 

An aspect of the present invention is a pattern 

25 extracting apparatus, wherein the model resolution 
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estimating processing unit estimates a resolution of 
which a luminance value of a color and a color 
difference registered in the color difference table 
optimally confirms to a local area of the input image 
as a read resolution of the print model of the local 
area of the input image. 

Thus, in an input image that is read by a device 
from a real printed document, for an area that exceeds 
a predetermined threshold value of the entire input 
image, the labeling process can be performed for the 
area with a threshold value optimum for the area. 
Consequently, the accuracy of the labeling process can 
be improved. 

An aspect of the present invention is a pattern 
extracting apparatus, wherein the model resolution 
estimation processing unit causes a read resolution 
of the print model to be a fixed value. 

Thus, the model resolution estimating process of 
a real printed document can be omitted. Consequently, 
the process can be performed at high speed. 

An aspect of the present invention is a pattern 
extracting apparatus, further comprising a grouping 
unit for categorizing label images labeled by the 
labeling unit as a group. 

Thus, a character area extracted by the labeling 
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process can be categorized as a character string area. 
Consequently, a title can be effectively extracted. 

An aspect of the present invention is a pattern 
extracting apparatus, wherein the grouping unit 
5 determines whether or not to categorize a label image 
as the same group corresponding to color information 
and geometry information of an enclosing rectangle of 
the label image . 

Thus, when a headline area with the same color is 

10 extracted from an input color image, a pattern is 
extracted from a limited area of the input color 
image. A pattern that is apart from a headline area 
can be removed from an object to be categorized. In 
addition, a small pattern such as noise and a large 

15 pattern such as a background that is not the headline 
characters can be omitted from an object to be 
processed. Consequently, the speed of the extracting 
process for the headline area can be improved. Since 
a labeled pattern is represented by an enclosing 

20 rectangle, a complicated pattern composing a character 
or the like can be represented in a simple shape. 
Thus, the storage capacity for the pattern can be 
decreased. In addition, when the relation of 
positions of patterns is obtained, the process can be 

25 simplified. 
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An aspect of the present invention is a pattern 
extracting apparatus, wherein the color information 
is the representative color of the pattern included 
in the enclosing rectangle. 
5 Thus, patterns labeled can be compared as 

enclosing rectangles. When patterns are categorized 
as a group corresponding to color information, it is 
not necessary to compare each pixel that composes the 
patterns. Thus, since the number of elements to be 

10 compared can be decreased, the process can be 
performed at high speed. 

An aspect of the present invention is a pattern 
extracting apparatus, wherein the geometry information 
is another enclosing rectangle present in a 

15 predetermined range of a considered enclosing 
rectangle . 

Thus, when a title area or the like is extracted 
from an input image, the range of an enclosing 
rectangle to be categorized as a group can be limited 
20 to a range suitable for the title area. The grouping 
process can be omitted for an area that protrudes from 
the title area. Thus, the process can be preformed 
at high speed. 

An aspect of the present invention is a pattern 
25 extracting apparatus, wherein the geometry information 
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calculating unit: has a rectangle number storing unit 
for storing a rectangle number corresponding to 
vertical and horizontal coordinates of each enclosing 
rectangle, a rectangle number extracting unit for 
5 extracting a rectangle number included in each of 
vertical and horizontal coordinates in a predetermined 
range of a considered enclosing rectangle, and an 
adjacent rectangle extracting unit for extracting a 
rectangle number included in both the vertical and 
10 horizontal coordinates as another enclosing rectangle 
in a predetermined range of a considered enclosing 
rectangle . 

Thus, another enclosing rectangle in the 
predetermined range of the considered enclosing 
15 rectangle can be effectively searched. 

An aspect of the present invention is a pattern 
extracting apparatus, wherein the rectangle number 
storing unit stores rectangle numbers of enclosing 
rectangles in the order of coordinate values, and 
20 wherein the rectangle number extracting unit searches 
rectangle numbers in the ascending order of the 
coordinate values and extracts rectangle numbers in 
each coordinate value. 

Thus, when an area for storing a rectangle number 
25 of an enclosing rectangle is prepared for each 
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coordinate value, the storage capacity corresponding 
to enclosing rectangles that are expectedly present 
in a predetermined range of a considered enclosing 
rectangle is required. In contrast, according to the 
5 present invention, since coordinate values 
corresponding to real enclosing rectangles are stored, 
the storage capacity can be decreased. 

An aspect of the present invention is a pattern 
extracting apparatus, wherein the determining unit 

10 categorizes two enclosing rectangles as the same group 
when the two enclosing rectangles are present in a 
predetermined range of a considered enclosing 
rectangle and the color difference of the 
representative colors of the two enclosing rectangles 

15 is smaller than a color difference obtained from the 
color difference table. 

Thus, when a title area or the like is extracted 
from an input image, a range of enclosing rectangles 
to be categorized can be limited to a range suitable 

20 for the title area. When enclosing rectangles are 
compared and patterns are categorized as a group with 
color information, the variation of a color in the 
reading operation of a real printed document can be 
considered. Thus, the process can be preformed at 

25 high speed. In addition, the reading accuracy is 
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improved . 

An aspect of the present invention is a pattern 
extracting apparatus, wherein the grouping unit 
categorizes a label image as a group corresponding to 
the thickness of a pattern of the label image. 
Thus, since characters with different thicknesses can 
be categorized as different groups, when a title or 
the like is extracted from an input image, only a 
character string composed of characters with the same 
thickness can be extracted as a title. Thus, the 
extracting accuracy of the title can be improved. 

An aspect of the present invention is a pattern 
extracting apparatus, wherein the grouping unit has 
a contour tracing unit for obtaining the contour 
length of the pattern, an area calculating unit for 
obtaining the area of the pattern, and a thickness 
calculating unit for obtaining the thickness of the 
pattern corresponding to the ratio of the area of the 
pattern and the contour length of the pattern. 

Thus, the thickness of the pattern can be 
accurately obtained. 

An aspect of the present invention is a pattern 
extracting apparatus, wherein the contour tracing unit 
searches a second pixel that is the contour of a 
pattern with a first pixel that is the contour of the 
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pattern in such a manner that the second pixel is 
searched from eight pixels that are adjacent to the 
first pixel and that are adjacent to a source pixel 
of the first pixel. 
5 Thus, to search a second pixel that becomes the 

contour of a pattern from a first pixel that becomes 
the contour of the pattern, eight pixels adjacent to 
the first pixel are searched in a predetermined 
direction. When a point of which the level of a pixel 

10 is changed from the level of a background to the level 
of the pattern is the second pixel, the first pixel 
that has already been extracted can be prevented from 
being extracted again as a point of the contour of the 
pattern. Thus, the speed of the contour tracing 

15 process can be improved. 

An aspect of the present invention is a pattern 
extracting apparatus, further comprising a character 
size estimating unit for estimating the character size 
corresponding to the size of an enclosing rectangle 

20 categorized as a group, and a rectangle deleting unit 
for deleting an enclosing rectangle that exceeds a 
predetermined range of the character size from the 
group of the enclosing rectangles. 

Thus, even if a pattern of a picture or the like 

25 is present in the vicinity of a title and the pattern 
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of "the picture is categorized as a group of characters 
that composes the title, the pattern of the picture 
can be removed from the group. Thus, the accuracy for 
extracting the title can be improved. 
5 An aspect of the present invention is a pattern 

extracting apparatus, further comprising a character 
string direction estimating unit for estimating the 
direction of a character string corresponding to the 
arrangement of enclosing rectangles categorized as a 
10 group by the grouping unit, and a character string 
extracting unit for extracting a character string 
corresponding to the arrangement of enclosing 
rectangles categorized as a group by the grouping 
unit. 

15 Thus, an area of which enclosing rectangles are 

irregularly disposed and an area of which enclosing 
rectangles are regularly disposed can be 
distinguished. Consequently, an area of which 
enclosing rectangles are regularly disposed can be 

20 treated as a title area. Thus, only the title area 
can be effectively extracted. 

An aspect of the present invention is a pattern 
extracting apparatus, further comprising a re-grouping 
unit for re-categorizing enclosing rectangles in a 

25 group corresponding to the size of the enclosing 



rectangles categorized as a group. 

Thus, characters with different sizes can be 
categorized as different groups. Consequently, when 
a title or the like is extracted from an input image, 
only a character string composed of characters with 
the same size can be treated as a title. Thus, the 
extracting accuracy for a title can be improved. 

An aspect of the present invention is a pattern 
extracting apparatus, further comprising an enclosing 
rectangle extracting unit for extracting an enclosing 
rectangle with a predetermined size from enclosing 
rectangles categorized as a group by the grouping 
unit, a horizontal arrangement flag assigning unit for 
assigning a horizontal arrangement flag to enclosing 
rectangles that are extracted as adjacent enclosing 
rectangles and whose upper edges or lower edges are 
close, a vertical arrangement flag assigning unit for 
assigning a vertical arrangement flag to enclosing 
rectangles that are extracted as adjacent enclosing 
rectangles and whose left edges or right edges are 
close, a character string direction determining unit 
for determining the direction of a character string 
of the group corresponding to the number of the 
horizontal arrangement flags and the vertical 
arrangement flags, and a character string determining 



32 

unit for determining whether the enclosing rectangles 
of the group are enclosing rectangles of a character 
string corresponding to the ratio of the enclosing 
rectangles of the group with the flags corresponding 
5 to the determined direction of the character string. 

Thus, it can be determined whether or not 
enclosing rectangles in a group are arranged in a row. 
A group of enclosing rectangles arranged in a row can 
be treated as a title area. Thus, a title can be 

10 effectively extracted from an input image. 

An aspect of the present invention is a pattern 
extracting apparatus, further comprising an enclosing 
rectangle generating unit for generating an enclosing 
rectangle categorized as a group by the grouping unit, 

15 and a representative color calculating unit for 
calculating the representative color of the group 
corresponding to the representative color of the 
pattern of the group categorized by the grouping unit. 
Thus, corresponding to all colors of a title area, 

20 the color of a title can be calculated. The influence 
of color difference in a local area of the title area 
can be alleviated. Consequently, the extracting 
accuracy for the title area can be improved. 

An aspect of the present invention is a pattern 

25 extracting apparatus, further comprising a group 
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combining unit for combining adjacent enclosing 
rectangles of the group corresponding to the 
representative color of the group. 

Thus, even if characters composing the same title 
5 are categorized as different groups due to a slight 
color difference, these characters can be categorized 
as the same group. Thus, a title area can be 
effectively extracted. 

An aspect of the present invention is a pattern 
10 extracting apparatus, wherein the group combining unit 
combines adjacent enclosing rectangles of a group when 
the difference of the sizes of the adjacent enclosing 
rectangles is in a predetermined range and the 
difference of representative colors of the group is 
15 in a predetermined range. 

Thus, when enclosing rectangles of a group are 
combined, only characters that compose the same title 
can be effectively combined. 

An aspect of the present invention is a pattern 
20 extracting apparatus, further comprising a re- 
extracting unit for re-extracting a pixel of an 
original image in the range of enclosing rectangles 
of the group corresponding to the representative color 
of the group. 

25 Thus, even if characters composing a title include 
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a sonant letter or a semi-sonant letter and the sonant 
letter or the semi-sonant is dropped from the group 
of the title due to a local color difference of the 
title area, the color of the sonant letter or the 
5 semi-sonant letter can be compared with the entire 
color of the title area so as to re-extract the sonant 
letter or the semi-sonant letter. Consequently, the 
extracting accuracy for the title can be improved. 

An aspect of the present invention is a pattern 

10 extracting apparatus, further comprising a digitizing 
unit for digitizing a pixel that has been re- 
extracted, a second labeling unit for assigning a 
label to the digitized image, and an adding unit for 
adding an enclosing rectangle that has been assigned 

15 a label by the second labeling unit and that is larger 
than a predetermined threshold value to the group. 

Thus, when a sonant letter or a semi-sonant letter 
is re-extracted from the title area, even if a thin 
noise with the same color as the title is present in 

20 the title area, only the noise can be deleted. 

Consequently, the extracting accuracy for the title 
can be improved. 

An aspect of the present invention is a pattern 
extracting apparatus, further comprising an output 

25 order determining unit for determining the output 
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order of patterns in a group corresponding to the 
areas of enclosing rectangles categorized as the 
group, the positions thereof, and the relation of the 
positions thereof . 
5 Thus, even if a plurality of title alternatives 

are extracted from an input image, the likelihood of 
a title can be evaluated. Consequently, title 
alternatives can be output in the order of the maximum 
likelihood of the title. 

10 An aspect of the present invention is a pattern 

extracting apparatus, further comprising a binary 
pattern generating unit for generating a binary 
pattern of which a label image of enclosing rectangles 
categorized as a group is assigned level 1. 

15 Thus, a binary image can be generated with a label 

image labeled with color information. Consequently, 
a pattern extracted from a color image can be 
effectively recognized. 

These and other objects, features and advantages 

20 of the present invention will become more apparent in 
light of the following detailed description of a best 
mode embodiment thereof, as illustrated in the 
accompanying drawings. 

25 Brief Description of Drawings 
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Fig. 1 is a schematic diagram showing a method for 
generating a conventional color separated image; 

Fig. 2 is a schematic diagram showing a 
conventional area expanding method; 
5 Fig. 3 is a block diagram showing the structure of 

a pattern extracting apparatus according to a first 
embodiment of the present invention; 

Fig. 4 is a block diagram showing the structure of 
a pattern extracting apparatus according to a second 
10 embodiment of the present invention; 

Fig. 5 is a block diagram showing the structure of 
a pattern extracting apparatus according to a third 
embodiment of the present invention; 

Fig. 6 is a block diagram showing the structure of 
15 a pattern extracting apparatus according to a fourth 
embodiment of the present invention; 

Fig. 7 is a block diagram showing the system 
structure of a pattern extracting apparatus according 
to an embodiment of the present invention; 
20 Fig. 8 is a block diagram showing the structure of 

a pattern extracting apparatus according to a firth 
embodiment of the present invention; 

Fig. 9 is a block diagram showing the structure of 
a pattern extracting apparatus according to a sixth 
25 embodiment of the present invention; 
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Fig. 10 is a flowchart showing the operation of 
the pattern extracting apparatus shown in Fig. 9; 

Figs. 11A and 11B are schematic diagrams for 
explaining a reduced image generating method according 
5 to an embodiment of the present invention; 

Figs. 12A and 12B are schematic diagrams for 
explaining a labeling method according to an 
embodiment of the present invention; 

Figs. 13A and 13B are schematic diagrams showing 
10 an example of the structure of a threshold value 
reference table according to an embodiment of the 
present invention; 

Fig. 14 is a schematic diagram showing another 
example of the structure of the threshold value 
15 reference table according to an embodiment of the 
present invention; 

Figs. 15A, 15B, and 15C are schematic diagrams for 
explaining an enclosing rectangle generating process 
and an original image label assigning process 
20 according to an embodiment of the present invention; 

Figs. 16A and 16B are schematic diagrams for 
explaining a representing method of a color image in 
halftone printing method; 

Fig. 17 is an enlarged view showing a color image 
25 in the halftone printing method; 
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Fig. 18A is a schematic diagram showing an area 
printed with one color by the halftone printing 
method ; 

Fig. 18B is a schematic diagram showing read 
meshes in the area shown in Fig. 18A; 

Fig. 18C is a schematic diagram showing read 
results of individual pixels; 

Figs, 19A to 19D are schematic diagrams showing a 
print model generating method according to an 
embodiment of the present invention; 

Fig. 20 is a schematic diagram showing the 
structure of a color difference table according to an 
embodiment of the present invention; 

Fig. 21 is a block diagram showing the structure 
of a color difference table generating apparatus 
according to an embodiment of the present invention; 

Fig. 22 is a block diagram showing the structure 
of a pattern extracting apparatus according to a 
seventh embodiment of the present invention; 

Fig. 23 is a flowchart showing the operation of 
the pattern extracting apparatus shown in Fig. 9; 

Fig. 24 is a schematic diagram for explaining a 
model resolution estimating process according to an 
embodiment of the present invention; 

Fig. 25 is a schematic diagram for explaining a 
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color labeling process according to an embodiment of 
the present invention; 

Fig. 26 is a schematic diagram for explaining an 
assigning method for a search range in the vicinity 
5 of a considered rectangle according to an embodiment 
of the present invention; 

Fig, 27 is a schematic diagram for explaining a 
rectangle extracting method in a search range in the 
vicinity of a considered rectangle according to an 
10 embodiment of the present invention; 

Figs. 28A, 28B, and 28C are schematic diagrams for 
explaining a storing method of rectangle information 
in a search range in the vicinity of a considered 
rectangle according to an embodiment of the present 
15 invention; 

Figs. 29A and 29B are schematic diagrams for 
explaining a calculating method of the thickness of 
a character line according to an embodiment of the 
present invention; 
20 Fig. 30 is a schematic diagram for explaining a 

re-grouping process corresponding to the thickness of 
a character line according to an embodiment of the 
present invention; 

Figs. 31A, 31B, 31C, and 31D are schematic 
25 diagrams for explaining an assigning method of a 
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search start point of a pixel in a contour tracing 
process according to an embodiment of the present 
invention; 

Fig. 32 is a schematic diagram showing a direction 
5 code viewed from a current pixel according to an 
embodiment of the present invention; 

Figs. 33A and 33B are schematic diagrams for 
explaining a process for deleting a large rectangle 
from a group according to an embodiment of the present 
10 invention; 

Figs. 34A and 34B are schematic diagrams for 
explaining a method for calculating the maximum 
frequency value of the height of rectangles according 
to an embodiment of the present invention; 
15 Figs. 35A and 35B are schematic diagrams for 

explaining a process for combining overlapped 
rectangles in a group according to an embodiment of 
the present invention; 

Figs. 36A and 36B are schematic diagrams for 
20 explaining a process for extracting a character string 
from a group according to an embodiment of the present 
invention; 

Figs. 37A and 37B are schematic diagrams for 
explaining a process for combining separated 
25 characters of a character string according to an 
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embodiment of "the present invention; 

Figs. 38A and 38B are schematic diagrams for 
explaining a re-grouping process corresponding to 
sizes of characters in a group according to an 
5 embodiment of the present invention; 

Figs. 39A and 39B are schematic diagrams for 
explaining a group combining process according to an 
embodiment of the present invention; 

Figs. 40A and 40B are schematic diagrams for 
10 explaining an extracting process for a rectangle and 
a group that is a likelihood of a character string 
according to an embodiment of the present invention; 

Fig. 41 is a schematic diagram for explaining a 
deleting process for a noise group according to an 
15 embodiment of the present invention; 

Figs. 42A and 42B are schematic diagrams for 
explaining a process for extracting patterns with same 
colors from a group according to an embodiment of the 
present invention; 
20 Fig. 43A is a schematic diagram showing an 

original image according to an embodiment of the 
present invention; 

Fig. 43B is a schematic diagram showing a label 
image of Fig. 43A; 
25 Fig. 43C is a schematic diagram showing a 
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considered rectangle assigned in the label image shown 
in Fig. 43B; 

Fig. 43D is a schematic diagram showing group 
extracted results; 

Fig. 44A is a schematic diagram showing results of 
which the group extracted results shown in Fig. 43D 
are re-grouped with the thickness of a character line; 

Fig. 44B is a schematic diagram showing results of 
which the overlapped rectangles shown in Fig. 44A are 
combined ; 

Fig. 44C is a schematic diagram showing results of 
which a rectangle and a group that is a likelihood of 
a character string are extracted from the image shown 
in Fig. 44B; 

Fig. 44D is a schematic diagram showing results of 
which a pattern in a group is extracted from the image 
shown in Fig. 44C; 

Fig. 45A is a schematic diagram showing a first 
alternative pattern of a title extracted from the 
image shown in Fig. 43 A; and 

Fig. 45B is a schematic diagram showing a second 
alternative pattern of the title extracted from the 
image shown in Fig. 43A. 

Description of Preferred Embodiments 
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Next, with reference to the accompanying drawings, 
a pattern extracting apparatus according to an 
embodiment of the present invention will be described. 

Fig . 3 is a block diagram showing the structure of 
a pattern extracting apparatus according to a first 
embodiment of the present invention. 

In Fig. 3, a color information calculating unit 1 
calculates color information of a pattern of an input 
image. A geometry information calculating unit 2 
calculates geometry information of a pattern of an 
input image. A categorizing unit 3 categorizes a 
pattern of an input image corresponding to the color 
information calculated by the color information 
calculating unit 1 and the geometry information 
calculated by the geometry information calculating 
unit 2. The color information is for example a color 
of a pattern in color space. The geometry information 
is for example the size of a particular pattern, the 
position of a particular pattern in an input image, 
or the relation between positions of a particular 
pattern and another pattern. 

When a pattern of an input image is categorized, 
the geometry information of the pattern is also used 
along with the color information thereof. Thus, in 
the case that a title area with the same color is 
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extracted from a color input image, in a limited range 
of the color input image, patterns with the same color 
and different labels can be categorized as one group. 
Patterns at particular positions or patterns with a 
5 particular size can be categorized as one group even 
if they are assigned different labels. 

Thus, only in a range that a title area is present 
with a high probability, the labeling process can be 
performed. In addition, a small pattern that is not 

10 a title character such as noise and a large pattern 
such as a background can be removed from an object to 
be processed. Thus, the speed of the extracting 
process for a title area can be improved. In 
addition, the influence of colors of the non-title 

15 areas can be alleviated. Consequently, the extracting 
accuracy of the title area can be improved. 

Fig. 4 is a block diagram showing the structure of 
a pattern extracting apparatus according to a second 
embodiment of the present invention. 

20 In Fig. 4, a clustering unit 11 clusters pixels of 

an input image corresponding to color information of 
adjacent pixels. A grouping unit 12 groups clusters 
corresponding to the color information and geometry 
information of each cluster obtained by the clustering 

25 unit 11. 
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Thus, when pixels of an input image are clustered, 
color information of a considered pixel is compared 
with color information of each of adjacent pixels. 
Thus, it is not necessary to compare color information 
of all pixels of the input image. 

When all the pixels of the input image are 
compared with each other, the number of times of the 
comparing process of color information of each pixel 
amounts to the square of the number of pixels of the 
input image. In contrast, when the color information 
of a particular pixel is compared with the color 
information of each of adjacent pixels, the number of 
times of the comparing process of color information 
of each pixel becomes the number of pixels of the 
input image. Thus, the clustering process can be 
performed at high speed. 

When clustered pixels are grouped, pixels in the 
same cluster can be integrally handled. Thus, it is 
not necessary to process individual pixels. 
Consequently, the grouping process can be performed 
at high speed. 

In addition, clustered pixels can be grouped in a 
limited range of the input image. Moreover, a small 
cluster that is noise can be excluded in the grouping 
process. Thus, the grouping process can be performed 
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at much higher speed. In particular, when the number 
of labels is large as with a color image, a remarkable 
effect can be obtained. 

Fig. 5 is a block diagram showing the structure of 
5 a pattern extracting apparatus according to a third 
embodiment of the present invention. 

In Fig. 5, a color difference calculating unit 21 
calculates the color difference of adjacent pixels in 
an area represented by a predetermined color. A 

10 threshold value assigning unit 22 assigns a threshold 
value corresponding to the color difference calculated 
by the color difference calculating unit 21. A 
labeling unit 23 labels pixels adjacent to a pixel 
represented by a predetermined color corresponding to 

15 the threshold value assigned by the threshold value 
assigning unit 22 . 

It is assumed that an input image is obtained from 
a document printed by the halftone printing method and 
that the colors of patterns of the input image are 

20 represented in a combination of the sizes of dots of 
basic colors. Since the sizes of dots of the basic 
colors are small, when one color is represented in a 
combination of the sizes of dots of the basic colors, 
even if the difference of colors of the dots cannot 

25 be distinguished by the naked eye, and viewed as a 
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single color, when the read resolution of a device is 
as high as the color of each dot can be distinguished, 
an area viewed as a single color by the naked eye may 
be determined as a non-single color area by the 
5 device. 

Thus, when the read results of the device are 
labeled, one pattern viewed as a single color by the 
naked eye is assigned different labels. When patterns 
extracted corresponding to the different labels are 

10 viewed by the naked eye, a hole or a brake takes place 
in an area recognized as one pattern with the naked 
eye. Thus, the extracting accuracy of the pattern 
deteriorates . 

To prevent such a problem, for an area that is 

15 viewed as a single color by the naked eye, the color 
variation read by the device is obtained. Pixels in 
the range of the color variation are assigned the same 
label. Thus, one pattern viewed as a single color by 
the naked eye can be assigned the same label. 

20 Consequently, a hole or a break can be prevented from 
taking place in an area viewed as one pattern by the 
naked eye. 

Fig. 6 is a block diagram showing the structure of 
a pattern extracting apparatus according to a forth 
25 embodiment of the present invention. 
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In Fig. 6, a first color information comparing 
unit 31 compares color information of adjacent pixels 
of an input image. A second color information 
comparing unit 32 compares color information of pixels 
5 compared by the first color information comparing unit 
31 and color information of a label image adjacent to 
the pixels. A labeling unit 33 labels pixels 
corresponding to the compared results of the first 
color information comparing unit 31 and the compared 

10 results of the second color information comparing unit 
32. Color information of the label image is for 
example the average value of color information of 
pixels with the same label. 

It is assumed that the color of a title area of an 

15 input image gradually varies and thereby the boundary 
of a character string of a title and a background 
thereof is not clear. In this case, if color 
information of adjacent pixels is compared, since 
colors of the adjacent pixels are similar, these 

20 pixels are assigned the same label. When these pixels 
are successively traced, a pattern of the background 
is read in the title area. Thus, pixels with 
different colors from the title may be assigned the 
same label . 

25 Rather than comparing color information of 
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adjacent pixels, color information of labeled patterns 
is compared with color information of each pixel. In 
addition, the pixel is assigned a label. Thus, even 
if the color gradually varies, the boundary between 
5 the title area and the background can be detected. 

Fig. 7 is a block diagram showing the system 
structure of a pattern extracting apparatus according 
to an embodiment of the present invention. 

In Fig. 7, the pattern extracting apparatus 

10 comprises a central processing unit (CPU) 41, a read- 
only memory (ROM) 42, a random-access memory (RAM) 43, 
a communication interface 44, a communication network 
45, a bus 46, an input/output interface 47, a display 
unit 48 that displays recognized results such as a 

15 character string of a title or the like, a printer 49 
that prints recognized results such as a character 
string of a title or the like, a memory 50 that 
temporarily stores data that is read by a scanner 51 
or the like, a scanner 51 that reads an input image 

20 or the like, a keyboard 52, a driver 53 that drives 
a storage medium, a hard disk 54, an IC memory card 
55, a magnetic tape 56, a floppy disk 57, and an 
optical disc 58 (such as a CD-ROM or a DVD-ROM). 

A program that causes the system to perform a 

25 pattern extracting process is stored in a storage 
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medium such as the hard disk 54, the IC memory card 
55, the magnetic tape 56, the floppy disk 57, or the 
optical disc 58. When the system reads the program 
for the pattern extracting process from such a storage 
5 medium to the RAM 43, the system performs the pattern 
extracting process. The program for the pattern 
extracting process may be stored in the ROM 42 . 

The program for the pattern extracting process can 
be obtained from the communication network 45 through 

10 the communication interface 44. Examples of the 
communication network 45 connected to the 
communication interface 44 are LAN (Local Area 
Network), WAN (Wide Area Network), Internet, analog 
telephone network, digital telephone network (ISDN: 

15 Integral Service Digital Network), PHS (Personal Handy 
System), and a radio communication network (such as 
satellite communication) . 

When the program for the pattern extracting 
process gets started, the CPU 41 assigns a threshold 

20 value with which an input image that is read by the 
scanner 51 is labeled and generates a label image 
corresponding to the input image that is read by the 
scanner 51. After generating the label image, the CPU 
51 generates enclosing rectangles of the pattern of 

25 the input image corresponding to the labels thereof. 
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The CPU 41 groups the enclosing rectangles 
corresponding to color information of areas therein 
and geometry information thereof. The CPU 41 extracts 
a title area or the like corresponding to the 
5 arrangement state and sizes of the grouped enclosing 
rectangles. When the CPU 41 extracts the title area, 
it outputs a pattern in the title area as a title 
extracted from the input image to the display unit 48 
and the printer 49. The CPU 41 recognizes characters 
10 of the pattern in the title area and automatically 
retrieves a document corresponding to the recognized 
results. 

Fig. 8 is a block diagram showing the structure of 
a pattern extracting apparatus according to a fifth 
15 embodiment of the present invention. 

In Fig. 8, the pattern extracting apparatus 
comprises a color image inputting unit 61 , an external 
outputting unit 62, a memory 63, a central processing 
unit (CPU) 64, an image reducing unit 65, a variable 
20 threshold value assigning unit 66, and a labeling unit 
67. 

When the image reducing unit 65 inputs a color 
image signal from the color image inputting unit 61, 
the image reducing unit 65 generates a reduced image 
25 of which the number of pixels of the original image 



52 

is decreased 1/9 times. The labeling unit: 67 labels 
the reduced image with a color difference to some 
extent using a technique such as the area expanding 
method and obtains a label image that represents the 
5 same label area and an enclosing rectangle that 
surrounds the same label area. 

At this point, the threshold value of the 
difference of color values with the same label as 
adjacent pixels is not fixed (unlike with the area 

10 expanding method), but variable depending on a 
relevant color. The threshold value is obtained 
corresponding to the relevant color by the variable 
threshold value assigning unit 66. The variable 
threshold value assigning unit 66 references a 

15 particular table corresponding to the relevant color. 

All numeric strings such as (R, G, B) that 
represent colors of areas with the same label are 
searched from the reduced image. The obtained numeric 
strings are stored as attributes of the label . In 

20 addition, the representative color of the label is 
obtained corresponding to the colors. 

Next, the enclosing rectangle obtained from the 
reduced image is reflected to the original image. 
Only the inside of the rectangle is scanned so as to 

25 detect a color similar to the representative color. 
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Pixels adjacent to the relevant pixel are assigned 
labels. In this case, colors treated as the same 
color are colors that have been treated as the same 
color by the above-described process, 
5 A predetermined area is extracted from a label 

image obtained from the original image. When a 
character area such as a title composed of large 
characters or a keyword is extracted from a color 
document, only a relevant character string area is 

10 extracted using character string extracting technique 
corresponding to information of the sizes and 
arrangement of enclosing rectangles. 

Fig. 9 is a block diagram showing the structure of 
a pattern extracting apparatus according to a sixth 

15 embodiment of the present invention. 

In Fig. 9, the pattern extracting apparatus 
comprises a color image inputting unit 71, an external 
outputting unit 72, a memory 73, and a central 
processing unit 74. 

20 The color image inputting unit 71 inputs an 

external color image as an RGB signal. Examples of 
the color image inputting unit 71 are a CCD camera or 
a scanner. 

The external outputting unit 72 outputs various 
25 results processed by the central processing unit 74. 
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An example of the external outputting unit 72 is a 
display unit. 

The memory 73 stores image information and 
threshold value information. For example, the memory 
5 73 stores a color image signal that is input from the 
color image inputting unit 71 and results that are 
processed by the central processing unit 74 for a 
stored color image signal. 

The central processing unit 74 generates a reduced 

10 image corresponding to a color image signal that is 
input from the color image inputting unit 71 and 
performs various processes such as a labeling process 
and a particular area extracting process. The central 
processing unit 74 comprises an image reducing unit 

15 75 , a variable threshold value assigning unit 76, a 
labeling unit 77, an enclosing rectangle process 78, 
an original image labeling unit 79, and a particular 
area extracting unit 80. 

Fig. 10 is a flowchart showing the operation of 

20 the color image processing apparatus shown in Fig. 7. 

In Fig. 10, an object such as a color document is 
input by the color image unit 71 that is a 
photographing unit such as a CCD camera or a scanner. 
The photographed color image is stored in the memory 

25 73 (at step SI). 
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Next, the image reducing unit 75 reduces the color 
image stored in the memory 73 with a fixed reduction 
ratio or a particular reduction ratio of which the 
desired size of an image is obtained. The reduced 
5 image is stored in the memory 73 (at step S2 ) . The 
reduction ratio is also stored. As the reducing 
method, as shown in Fig. 11A, when a color image is 
reduced 1/3 times, an area (3 x 3) of the original 
image represented by (R, G, B) is assigned to a 

10 reduced image (lxl). In this case, with (R, G, B) 
of colors of nine pixels of the area (3 x 3) of the 
original image, a histogram as shown in Fig. 11B is 
generated. The most frequent value is assigned as the 
value of the relevant pixel of the reduced image. 

15 If there are a plurality of values as the most 

frequent value, one of the plurality of values is 
selected as the value of the relevant pixel of the 
reduced image corresponding to a predetermined 
algorithm . 

20 Generally, when an area (n x n) of an original 

image is assigned to a reduced image (lxl) (namely, 
the original image is reduced 1/n times), with (R, G, 
B) of colors of pixels of the area (n x n) of the 
original image, a histogram is generated. For 

25 example, an intermediate value or the most frequency 
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value of the histogram is used as the value of the 
relevant pixel of the reduced image. 

A value assigned to a reduced image is not a 
created value such as the average value of colors of 
5 pixels of the area (n x n) of the original image, but 
a value of the original image* 

Thereafter, the labeling unit 77 performs a 
labeling process for the reduced image. Namely, the 
labeling unit 77 assigns the same label to adjacent 

10 pixels with similar colors of the reduced image (at 
step S3). In other words, the labeling unit 77 
raster-scans the reduced image from the upper left 
position and assigns a non-labeled pixel a label value 
of the maximum label value plus 1 . 

15 For example, as shown in Fig. 12A, assuming that 

a color value of a considered pixel C is (Rc, Gc, Be), 
that eight adjacent pixels of the considered pixels 
C are denoted by 1 to 8, and that color values of the 
eight adjacent pixels are denoted by (Ri, Gi, Bi) 

20 (where i = 1 to 8 ) , the color values (Ri, Gi, Bi) of 
these pixels are obtained. In other words, the color 
values (RI, GI, BI ) , (R2, G2, B2), ... (R8, G8, B8 ) 
of the pixels 1, 2, ... 8 are obtained. 

The distances dci (where i = 1 to 8 ) between the 

25 color value (Rc, Gc, Be) of the considered pixel and 
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the color value (Ri, Gi, Bi ) of each of the eight 
adjacent pixels are obtained. When the distance dci 
is smaller than a threshold value (Rth, Gth, Bth) with 
respect to R, G, B, the relevant pixel is assigned the 
same label as the considered pixel. For example, when 
the distance del between the pixel 1 and the 
considered pixel C is obtained, it is determined 
whether or not R, G, and B of the pixel 1 are smaller 
than Rth, Gth, and Bth, respectively. When the value 
of the three colors is smaller than the threshold 
value of the three colors (namely, the color of the 
pixel 1 is similar to the color of the considered 
pixel C), a small value is assigned to the distance 
dci so as to assign the same level as C to the pixel 
i. However, when at least one of three colors of the 
pixel 1 is larger than the threshold value of the 
three colors, a large value (larger than the threshold 
value th) is assigned to the distance del so as to 
assign a label different from the considered pixel C. 
The distance del is obtained as follows. 

del = |Rc - Rl | < Rth 

and | Gc - Gl | < Gth 

and | Be - Bl | < Bth . . . (1) 

When the formula ( 1 ) is satisfied, the same label 
is assigned as del = 0. When the formula (1) is not 
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satisfied, a value larger than the threshold value is 
assigned as del = 1000. Generally, as shown in Fig. 
12B, the distances dci (where i - 1 to 8 ) can be 
represented by the following formula. 
5 0 if |Rc - Ri | < Rth and 

| Gc - Gi | < Gth and 
| Be - Bi | < Bth 
1000 otherwise 

Note that the assigned value is not limited to 

10 1000. Instead, any value larger than the threshold 
value can be used. 

The threshold value (Rth, Gth, Bth) of three 
colors are obtained from a table that has been 
obtained from samples that depend on both an inputting 

15 unit (such as a CCD camera or a scanner) and an 
object. In addition, with a key of the color value 
(R, G, B) of the considered pixel, the threshold value 
is pre-assigned corresponding thereto. 

For example, as shown in Fig. 13A, with reference 

20 to a table having entries of all combinations of color 
values (R, G, B) of predictable considered pixels of 
input images, a threshold value (Rth, Gth, Bth) of 
three colors is obtained. In the table shown in Fig. 
13A, when the color value of a considered pixel is 

25 (Rl, Gl, Bl), the threshold value of G, G, B is (Rthl, 
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Gthl, Bthl ) . When the color value of a considered 
pixel is (R2, G2, B2 ) , "the threshold value thereof is 
( Rth2 , Gth2 , Bth2 ) . 

In the case shown in Fig. 13A, a threshold value 
reference table is used. The threshold value 
reference table has key entries of combinations of all 
color values of a considered pixel predicted from a 
distribution of a sample. With a combination of the 
color of the considered pixel, the threshold value 
reference table is referenced. Thus, the size of the 
table depends on the number of combinations of 
individual colors. In other words, the size of the 
table becomes very large. 

To prevent the threshold value reference table 
from becoming large, a table having only 
representative entries clustered with (R, G, B) can 
be used. In this case, the distance value between the 
color value (R, G, B) of a considered pixel and color 
value of a representative entry is obtained ( in other 
words, the similarity between the considered pixel and 
the representative entry is obtained ) . The closest 
representative entry is extracted. With a key of the 
closest representative entry, the threshold value 
(Rth, Gth, Bth) of three colors can be obtained. 

Alternatively, to prevent the threshold value 
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reference table from becoming large, values of 
individual colors are represented in N levels ( for 
example, N = 256). Threshold value reference tables 
are prepared for individual colors. Threshold values 
5 for individual colors are obtained from respective 
threshold value reference tables. In this case, the 
size of the threshold value reference tables becomes 
as small as 256 x 3 entries. For example, when the 
color value of a considered pixel is (Rl, G2, B2), as 

10 shown in Fig. 13B, as a threshold value for Rl , a 
threshold value table R is referenced and thereby Rthl 
is obtained. As a threshold value for G2, a threshold 
value table G is referenced and thereby Gth2 is 
obtained. As a threshold value for B2, a threshold 

15 value table B is referenced and thereby Bth2 is 
obtained . 

As a format of a threshold value reference table 
for obtaining a threshold value, as shown in Fig. 14, 
color values are categc>rized as a plurality of groups 

20 and threshold values are obtained for each group 
unlike with all combinations of (R, G, B) shown in 
Fig. 13A. In Fig. 14, one threshold value is assigned 
every four values of each color. Rl to R4 are 
assigned the same threshold value Rtl. R5 to R8 are 

25 assigned the same threshold value Rt2. Likewise, Gl 
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to G4 are assigned the same threshold value Gtl . BN-3 
to BN are assigned the same threshold value BtM. 

When an object is a printed document and input by 
a scanner, such a threshold value reference table can 
5 be automatically generated with a model of the printed 
document . 

Next, the enclosing rectangle processing unit 78 
performs an enclosing rectangle processing process for 
the labeled reduced image (namely, the label image) 

10 (at step S4). In other words, at step S3, as the 
results of the labeling process for the reduced image, 
the same area is assigned the same label value. For 
the same area, an enclosing rectangle is obtained. 

For example, as shown in Fig. 15A, when a color 

15 image 91 with character areas 93 to 95 and another 
area 92 (the color of the character areas 93 to 95 is 
different from the color of the area 92), a reduced 
label image 96 as shown in Fig. 15B is generated. 
Thus, enclosing rectangles 98 to 100 can be obtained 

20 with the reduced label image 96. 

In Fig. 15B, the rectangle area 97 corresponds to 
the background area 92 shown in Fig. 15A. The 
rectangle areas 98 to 100 correspond to the character 
areas 93 to 95 shown in Fig. 15A, respectively. 

25 In the label image, pixels recorded with the same 
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labels are scanned. All the colors of the pixels are 
recorded as attribute information of individual 
enclosing rectangles. One representative color of 
each label is obtained from all the pixels recorded 
5 with the same labels in the label image. The 
representative color can be obtained in the following 
manner. The frequency distribution of each color is 
obtained. A color with the highest frequency is 
treated as the representative color. The 
10 representative color information is recorded as 
attribute information of the relevant enclosing 
rectangle. 

The labeling unit 77 generates the following 
output information as results of the labeling process. 

15 As the label image that is two-byte information 

per pixel, the height , width, and reduction ratio of 
the reduced image area are represented. In addition, 
the number of rectangles (nr) in the reduced image 
area and rectangle information are represented. As 

20 the rectangle information, the number of rectangles 
(nr) in the area, a rectangle number, a label number 
of a label image in the rectangle, rectangle upper 
left coordinates (xl, yl ) , rectangle lower right 
coordinates (x2, y2 ) , the representative color (Rs, 

25 Gs, Bs) in the rectangle, the number of colors (nc), 
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colors (RO, GO, BO), (Rl, Gl, Bl ) , ... (Rn-1, Gn-1, 
Bn-1) are output:. 

The original image labeling unit 79 receives the 
output information, divides the upper left coordinate 
5 value and the lower right coordinate value of the 
rectangle information by the reduction ratio, and 
obtains the relevant coordinates in the original 
image. When the reduction ratio is 1/n, the upper 
left coordinates and the lower right coordinates of 

10 the rectangle of the original image are (xl * n, yl 
* n) and {(x2 * n) - 1, (y2 * n) - 1}, respectively 
(where * represents a multiplication and "-1" is 
subtracted from each of the x axis and the y axis of 
the lower right coordinates ) . 

15 The original image labeling unit 79 raster- scans 

the inside of the rectangle represented by the 
coordinates and searches a non-labeled pixel with a 
color similar to the representative color (Rs, Gs, 
Bs). When the color of a particular pixel is similar 

20 to the representative color, the following conditions 
are satisfied. 

| Rs - Ri | < Rth 
and 

| Gs - Gi | < Gth 
25 and 
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|Bs - Bi | < Bth 
In this case, the threshold value Rth, Gth, and Bth 
is fixed. 

When the original image labeling unit 79 detects 
5 a pixel that satisfies the above-described conditions, 
it assigns the pixel a label value of the maximum 
label value plus 1 . 

Thereafter, the original image labeling unit 79 
scans eight pixels adjacent to the considered pixel 
10 and obtains the color value (Ri, Gi, Bi ) of each of 
the adjacent pixels. When the color value (Ri, Gi, 
Bi) of each adjacent pixel and the color value (Rc, 
Gc, Be) of the considered pixel satisfy the following 
conditions, the original image labeling unit 79 assign 
15 the relevant pixel the same label as the considered 
pixel ( at step S5 ) . 

| Rc - Ri | < Rth 

and 

| Gc - Gi | < Gth 
20 and 

| Be - Bi | < Bth 

When the original image labeling unit 79 does not 
assign the relevant pixel the same label, it compares 
the color of the relevant pixel with all colors as the 
25 attribute information of the considered rectangle and 
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obtains the distance thereof. When the distance is 
smaller than the threshold value, the original image 
labeling unit 79 assigns the relevant pixel the same 
label as the considered pixel. 
5 In such a manner, as shown in Fig. 15C, the 

original image labeling unit 79 assigns labels to 
pixels with colors similar to the representative 
colors in the rectangles 97 to 100 and extracts areas 
102 to 105 with the same color value as the rectangle 

10 areas 97 to 100. In a real color document, when an 
image area is not coarse unlike with the case shown 
in Fig. 15A (namely, the image area has a background 
and color characters with various sizes), the same 
label area becomes complicated unlike with the case 

15 shown in Fig. 15C. 

The particular area extracting unit 80 extracts a 
particular area from the label area 101 obtained from 
the original image (at step S6). When a character 
area that is a keyword such as a headline or a title 

20 is extracted from a color document, the particular 
area extracting unit 80 can extract a relevant 
character string area corresponding to a conventional 
character string extracting method using given 
information of the size and arrangement of an 

25 enclosing rectangle. 
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For example, the particular area extracting unit 
80 obtains the relation of adjacent rectangles with 
the extracted rectangle information and extracts 
adjacent rectangles whose coordinate values are close 
5 to each other (namely, the difference between the 
coordinate values is smaller than the threshold value ) 
and whose color difference of (R, G, B) is smaller 
than the threshold value as a character string. The 
external outputting unit 72 displays the extracted 

10 character string. 

When one object is extracted from a color scenery 
image, the representative colors of adjacent areas of 
the label image are converted into information of such 
as HSV (Hue, Saturation, Visibility). Areas with 

15 similar hues are combined. Thus, separated areas due 
to a shadow can be combined to one area. An area with 
a particular size can be output as an alternative area 
of an object. 

Thus, according to the sixth embodiment of the 

20 present invention, areas with similar colors are 
obtained from a reduced image. The obtained areas are 
precisely extracted from the original image. 
Consequently, the process time can be remarkably 
reduced. In addition, only adjacent pixels are 

25 labeled with a threshold value corresponding to colors 
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of the pixels. Consequently, pixels can be locally 
and precisely color-clustered. Thus, even if the 
color of an area to be extracted is similar to the 
color of a background thereof, they can be 
5 distinguished. Thus, an area can be accurately 
extracted . 

When a threshold value corresponding to the color 
of a considered pixel is obtained, a table with a key 
of three color elements of a considered pixel is 

10 referenced. Thus, the number of calculations for 
obtaining the threshold value is decreased. 

When a threshold value corresponding to the color 
of a considered pixel is obtained, the similarity of 
the distance between three color elements of the 

15 considered pixel and three color elements of a 
particular color of an entry of the table is obtained. 
With an entry most similar to the color of the 
considered pixel, the threshold value of the entry of 
the table is obtained. Thus, the size of the table 

20 that stores the threshold values can be decreased. 

In addition, a threshold value of three color 
elements is obtained from a table that stores a 
distribution of a sample that depends on both an 
inputting unit such as a CCD camera or a scanner and 

25 an object. Thus, since the table stores a threshold 
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value that depends on an inputting unit, an input 
image is accurately labeled. 

Next, a method for assigning a threshold value in 
the labeling process according to an embodiment of the 
5 present invention will be described. 

Color documents are printed in various printing 
methods such as gravure printing method. However, 
most color documents are printed in halftone printing 
method. When a document printed by the halftone 
10 printing method is enlarged, a color moire pattern 
takes place in an area seen as an equal color by the 
naked eye. 

Figs. 16A and 16B are schematic diagrams showing 
a color moire pattern that takes place in a document 
15 printed by the halftone printing method. 

In Fig. 16A, an area ("no" in Japanese) of an 

original image 111 is printed with a single color of 
orange. However, as shown in Fig. 16B, when a part 
of the area is enlarged, it is clear that the 

20 area is composed of various colors. 

Thus, when the color difference (the luminance 
difference of RGB) of adjacent pixels is compared with 
a fixed threshold value and it is determined whether 
or not these pixels are assigned the same label, a 
25 particular area may not be extracted. For example, 
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pixels 112 to 115 in the area "C" shown in Fig. 16B 
are assigned a label different from labels of the 
other pixels. Thus, when a character "<D" is 
extracted from the original image 111, since the 
5 pixels 112 to 114 are dropped, the extracting accuracy 
deteriorates . 

With a print model, the average luminance value of 
RGB and the luminance difference of RGB of adjacent 
pixels for the read resolution of each scanner are 

10 obtained and stored in a color difference table. With 
reference to the color difference table, the threshold 
value of each color is controlled so as to perform the 
labeling process. 

For example, when the color difference between the 

15 pixels 114 and 115 in the area is the maximum, 

the color difference thereof is assigned for a 
threshold value for extracting an area in orange. 
Thus, when the character "<D" is extracted from the 
original image 111, the pixels 112 to 114 can be 

20 assigned the same label as other pixels in the area 
"0" . Consequently, the pixels 112 to 114 can be 
prevented from being dropped. 

Next, a method for creating a color difference 
table that stores the average value of colors, color 

25 differences of adjacent pixels, and read resolutions 



of scanners will be described. The color difference 
table is created with color print models and scanner 
read models according to an embodiment of the present 
invention . 

5 Fig, 17 is an enlarged view showing a document 

printed by the color halftone printing method. 

In Fig. 17, in the halftone printing method, cyan 
(sky blue), magenta (purple), and yellow dots 121 to 
123 whose sizes vary at each position are regularly 

10 arranged in a mesh pattern. 

In the halftone printing method, colored circles 
of the first color of three primary colors (or four 
colors including black) are printed at predetermined 
mesh intervals so that a desired concentration is 

15 obtained. Next, with a rotated mesh, colored circles 
of the second color are printed with a different size 
at the same mesh intervals as* those of the first 
color. Thereafter, with another rotated mesh, colored 
circles of the third color are printed with a 

20 different size at the same mesh intervals as those of 
the first color. In this printing method, the printed 
circles can be seen as an equal color by the naked 
eye. 

Figs. 18A, 18B, and 18C are schematic diagrams for 
25 explaining the reason why luminance values of 
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case that one color is printed with one mesh point by 
the halftone printing method. 

In Fig. 18A, when one color is printed by the 
halftone printing method, colored circles Dl with a 
radius rl are printed at mesh points of a print mesh 
Ml. When a print color is changed in the halftone 
printing method, the radius rl of the colored circles 
Dl is changed. Thus, in areas of which another color 
is printed by the halftone printing method, the 
colored circles Dl with the radius rl are dispersedly 
disposed. 

When a printed document shown in Fig. 18A is read 
by a scanner or the like, an image of the printed 
document is read with a fixed resolution of the 
scanner or the like. Thus, as shown in Fig. 18B, the 
colored circles Dl are read as pixels sectioned by a 
read mesh M2 corresponding to the resolution of the 
scanner or the like. Since the colored circles Dl are 
dispersedly disposed, the areas of the colored circles 
Dl included in the pixels sectioned with the read mesh 
M2 are not equal. Thus, as shown in Fig. 18C, the 
luminance of a considered pixel at the center of the 
mesh M2 is different from the average luminance of 
eight pixels PI to P8 adjacent to the considered 
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pixel. 

On the other hand, when areas with the same color 
are extracted from the printed document shown in Fig, 
18A, since the document showing in Fig. 18A is printed 
with one color by the halftone printing method, the 
eight pixels PI to P8 adjacent to the considered pixel 
should be extracted as pixels with the same color as 
the considered pixel. Thus, the luminance difference 
between the considered pixel and each of the pixels 
PI to P8 is obtained and the maximum value of the 
luminance difference (in the example shown in Fig- 
18C, the luminance difference between the considered 
pixel and the pixel P4) is recorded. When areas with 
the same color are extracted from an input image, the 
maximum value of the luminance difference between the 
considered pixel and each of pixels adjacent thereof 
is used as a threshold value. Adjacent pixels with 
luminance differences that is not larger than the 
threshold value are extracted as pixels with the same 
color as the considered pixel. 

Next, a method for structuring a scanner read 
model that precisely corresponds to the halftone 
printing method will be described. 

Figs. 19A, 19B, 19C, and 19D are schematic 
diagrams showing a scanner read model corresponding 
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to the halftone printing method. 

In Figs, 19A, 19B, 19C, and 19D, print meshes Mil, 
M12, and M13 for cyan, magenta, and yellow are rotated 
and combined as a print model. To adjust the 
concentrations of cyan, magenta, and yellow, the sizes 
of colored circles Dll, D12, and D13 at mesh points 
of the print meshes Mil, M12, and M13 are adjusted. 

At this point, parameters of a scanner read model 
are as follows. 

- Mesh distance md of print meshes Mil, M12, and 
M13 (however, the mesh distance md is in common with 
yellow, cyan, and magenta) 

- Angle 011 to horizontal line of yellow mesh Mil 

Angle 012 to horizontal line of magenta mesh 

M12 

- Angle 013 to horizontal line of cyan mesh M13 
Radius rll of colored circle Dll of yellow 

- Radius rl2 of colored circle D12 of magenta 
Radius rl3 of colored circle D13 of cyan 

However, in this embodiment, to avoid complicity, 
parameters are assigned the following values, not 
variables . 

- Angle 011 to horizontal line of yellow mesh Mil 
= 15 degrees 

Angle 012 to horizontal line of magenta mesh 



74 

M12 = 30 degrees 

Angle 013 to horizontal line of cyan mesh M13 
= 45 degrees 

- Mesh distance md « 20 dots 
5 With the print model, the average RGB value (Rm, 

Gm, Bm) that is affected by the radiuses rll, rl2, and 
rl3 of the colored circles Dll, D12, and D13 of 
yellow, magenta, and cyan is simulated. In reality, 
a unit area that is much larger than the mesh distance 
10 md of the print meshes Mil, M12, and M13 is considered 
as follows . 

Rm = 255 - area of cyan in unit area x 255 / unit 

area 

Gm = 255 - area of magenta in unit area x 255 / 
15 unit area 

Bm = 255 - area of yellow in unit area x 255 / 
unit area 

Next, a scanner read model of which an image 
equally printed in one color (Rm, Gm, Bm) is read by 

20 a scanner will be described. In this model, a read 
mesh M14 with a distance sd that is different from the 
mesh distance of the print meshes Mil, M12, and M13 
is introduced. Luminance of RGB in the read mesh M4 
(namely, areas of individual colors) is obtained. 

25 In this case, the distance sd of the read mesh M14 
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is equivalent to the read resolution of the scanner. 
When an image with an average color (Gm, Gm, Bm) is 
read, if the read resolution of the scanner is high, 
the luminance values of RGB of adjacent pixels largely 
5 vary. In contrast, when the read resolution of the 
scanner is low, the colors of read pixels are close 
to the average color (Rm, Gm, Bm). The luminance 
difference of RGB of the adjacent pixels approaches 
to 0. 

10 The luminance value of RGB (Rc, Gc, Be) in the 

read mesh M14 is as follows. 

Rc = 255 - area of cyan in area of read mesh x 255 
/ area of read mesh 

Gc = 255 - area of magenta in area of read mesh x 
15 255 / area of read mesh 

Be = 255 - area of yellow in area of read mesh x 
255 / area of read mesh 

The luminance value of RGB (Rc, Gc, Be) of the 
read mesh M14 with the mesh distance sd is obtained. 
20 The luminance difference between the luminance value 
(Rc, Gc, Be) and the luminance value of each of 
adjacent pixels is obtained. A proper value (for 
example, the maximum value) is recorded to a color 
difference table. 
25 In a print model, colored circles at mesh points 
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overlap for each color of RGB. Thus, these colored 
circles mutually affect each other corresponding to 
the luminance value of RGB, Consequently, strictly 
speaking, it cannot be said that the print model is 
5 independent in RGB. However, in the scanner read 
model, for simplicity, it is supposed that the model 
is independent in RGB. 

Thus, RGB luminance value, read resolution, and 
luminance difference between a considered pixel and 
10 each of adjacent pixels are stored in tables that are 
independent in RGB. 

Fig. 20 is a table showing an example of a table 
for R. 

In Fig . 20 , the maximum value of luminance 
15 differences to adjacent pixels is listed corresponding 
to R luminance values (0 to 255) and scanner read 
resolutions (14 to 35). In the table shown in Fig. 
20, R luminance values are recorded at intervals of 
three. Any R luminance value in the range from 0 to 
20 255 can be obtained by an interpolating process. 

The table for R shown in Fig. 20 applies to tables 
for G and B. 

When a print model corresponding to the halftone 
printing method is created, colors created in the 
25 halftone printing method can be reproduced by a 
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computer. Thus, without; need to analyze a document 
printed in the halftone printing method, tables for 
RGB can be created. 

Referencing the tables for RGB, a threshold value 
5 necessary for detecting the range of the same color 
can be obtained with an input image. 

Fig. 21 is a block diagram showing the structure 
of a color difference table generating apparatus 
according to an embodiment of the present invention. 

10 In Fig. 21, the color difference table generating 

apparatus comprises a color image inputting unit 111, 
an external outputting unit 112, a memory 113, and a 
central processing unit (CPU) 114. The central 
processing unit 114 has a print model generating unit 

15 115 and a color difference table generating unit 116. 

The print model generating unit 115 generates a print 
model in the method shown in Fig. 19 and reproduces 
colors generated in the halftone printing method on 
a computer. The color difference table generating 

20 unit 116 generates a scanner read model with a print 
model corresponding to the halftone printing method 
and generates color difference tables for individual 
colors of RGB as shown in Fig. 20. 

Fig. 22 is a block diagram showing the structure 

25 of a pattern extracting apparatus according to a 
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seventh embodiment of the present invention. 

In Fig, 22, the pattern extracting apparatus 
according to the seventh embodiment comprises a color 
image inputting unit 131, an external outputting unit 
5 132, a memory 133, and a central processing unit (CPU) 
134. The central processing unit 134 has a color 
difference table 135, a color separated image 
generating unit 136, and a character area extracting 
unit 137. 

10 The color difference table 135 is a table shown in 

Fig. 20. The color difference table 135 stores 
luminance values, read resolutions, and luminance 
differences of adjacent pixels of each color of RGB. 
The color separated image generating unit 136 

15 searches the color difference table corresponding to 
the read resolution and RGB luminance value of an 
input image and obtains luminance differences of 
adjacent pixels corresponding to the read resolution 
and RGB luminance value of the input image. The 

20 luminance differences of the adjacent pixels are used 
as a threshold value for labeling the adjacent pixels. 
When the read resolution and RGB luminance value of 
the input image vary, with reference to the color 
table 135, the threshold value for labeling the 

25 adjacent pixels is updated. 
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With key information of the color (R, G, B) of a 
considered pixel, the color difference table 135 is 
searched for a threshold value of luminance 
differences of adjacent pixels treated as the same 
5 color. The obtained threshold value is applied to the 
adjacent pixels of the considered pixel. When the 
luminance difference between each adjacent pixel and 
the considered pixel for each color of RGB is not 
larger than the threshold value, the same label is 
10 assigned to the considered pixel and the adjacent 
pixels. 

In the labeling process, only pixels adjacent to 
a considered pixel are processed. In addition, pixels 
are assigned labels with threshold values 
15 corresponding to colors of the pixels. Thus, an input 
image can be locally and precisely color-clustered. 
Consequently, a label image free from holes and 
unsmooth contour can be generated. 

In addition, a character area with an equal color 
20 can be extracted quickly and accurately from a color 
document image . 

From an image that has a background and an object 
whose colors are locally similar and that has an area 
whose color is similar to the color of another area, 
25 areas whose colors are locally and accurately the same 
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can be extracted. 

In areas whose colors are treated as the same 
color, even if the color value of a pixel is slightly 
different from the color value of a pixel adjacent 
5 thereto, the area can be prevented from being 
extracted as a hole and an unsmooth contour. Thus, 
the area can be accurately extracted. 

When a label image is generated, enclosing 
rectangles of patterns of an input image are generated 
10 corresponding to labels of the patterns. The 
enclosing rectangles are grouped corresponding to 
color information of areas of the enclosing rectangles 
and geometry information thereof. 

The character area extracting unit 137 extracts a 
15 character string area from the label image obtained 
from the original image corresponding to the 
arrangement and sizes of the enclosing rectangles 
grouped by the color separated image generating unit 
136. 

20 When a character area as a keyword such as a 

headline or a title described with a large font is 
extracted from a color document, only the character 
area is extracted with information of the size and 
arrangement of enclosing rectangles. 

25 Fig. 23 is a flowchart showing the operation of 
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the pattern extracting apparatus shown in Fig. 22. 

In Fig. 23, an image inputting process is 
performed (at step Sll). In the image inputting 
process, a color image is input with a photographing 
5 unit such as a CCD camera or a scanner and stored in 
a memory . 

Thereafter, a model resolution estimating process 
is performed (at step S12). In the model resolution 
estimating process, the average luminance value and 

10 the maximum value of the luminance difference values 
of adjacent pixels registered in a color difference 
table obtained from a print model representing a moire 
pattern of a color printed document and a scanner read 
model thereof are compared with the average luminance 

15 value and the maximum value of the luminance 
difference values of adjacent pixels of the input 
image, respectively. Thus, an optimum scanner read 
resolution for the entire input image is obtained. 

Fig. 24 is a schematic diagram for explaining a 

20 model resolution estimating process according to an 
embodiment of the present invention. 

In Fig. 24, an input image 141 is divided into (3 
x 3) pixel blocks corresponding to a mesh 142. The 
average RGB luminance value (Rm, Gm, Bm) of (3 x 3) 

25 pixels a to i and the maximum value (Rd, Gd, Bd) of 
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luminance difference values of adjacent pixels are 
obtained for each block of the entire input image 141. 
There are 20 adjacent directions in the (3x3) pixels 
a to i. The luminance difference values in the 20 
5 adjacent directions of the pixels are obtained. The 
maximum value (Rd, Gd, Bd) of the luminance difference 
values is used. 

With key information of the average RGB luminance 
value (Rm, Gm, Bm) and the maximum value (Rd, gd, Bd) 

10 of the luminance difference values of the adjacent 
pixels, the color difference table is searched for a 
scanner read resolution value that is optimum for the 
average luminance value and the maximum value of the 
luminance difference values of the adjacent pixels 

15 registered in the color difference table. When the 
difference between the maximum value (Rd, Gd, Bd) of 
the luminance difference values of the adjacent pixels 
obtained from the input image 141 and the maximum 
value of the luminance difference values of the 

20 adjacent pixels with the scanner read resolution 
optimum in the color difference table is not in a 
predetermined range, a reject code (-1) is returned 
to the (3x3) pixels a to i. Otherwise, a conformity 
code (0) is returned. 

25 Such a process is performed for the entire input 
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image 141. Thus, the reject code (-1) or conformity 
code (0) is assigned all (3 x 3) pixels a to i of all 
the blocks of the mesh 142. For pixels assigned the 
conformity code (0) in the (3x3) pixels a to i, the 
5 average value of the scanner read resolutions thereof 
is obtained. The average value is treated as the 
scanner read resolution of the input image 141 . 
Next, a process for R will be exemplified. 
The resolution value of the table for R is fixed 

10 to a particular value. R luminance values of the 
table for R are scanned for an R luminance value Ri 
that satisfies Ri ^ Rm < Ri + 1 (where Rm is the 
average luminance value). At this point, assuming 
that the maximum value of the luminance values of 

15 adjacent pixels against the R luminance value Ri is 
Rdi and that the maximum value of the luminance values 
of the adjacent pixel against the R luminance value 
Ri + 1 is Rdi + 1, the average luminance value Rm is 
obtained by linearly interpolating the R luminance 

20 value Ri and the R luminance value Ri + 1 . The 
relation is applied to the maximum value of luminance 
values of the adjacent pixels so as to obtain the 
maximum value inf ered_delta of luminance values 
against estimated adjacent pixels. In other words, 

25 the following relation is satisfied. 
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rl = Ri - Ri 
rm = Rm - Ri 
rn = Ri + 1 - Rm 
inf ered_delta 

= Rri * rn / rl + Rri + 1 * rm / rl 

The difference delta_r between the estimated 
maximum value inf ered_delta of the luminance values 
against the adjacent pixels and the maximum value Rd 
of the luminance values obtained from the image is 
obtained as follows. 

delta_r = | Rd - inf ered__delta | 
The same process is performed for G and B so as to 
obtain delta_g and delta_b. The sum delta is obtained 
as follows . 

delta = delta_r + delta_g + delta_b 

The resolution parameters are scanned so as to 
obtain a resolution resol with the minimum delta value 
and a delta value delta_min. When the value delta_min 
satisfies the following condition: 

delta_min > TH_SUB (where TH_SUB is a fixed 
threshold value), 

it is determined that the relevant (3x3) block does 
not conform with the model. Thus, the reject code (- 
1) is returned. Otherwise, the conformity code (0) 
is returned. For only blocks assigned the conformity 
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code, the average value of the resolution values resol 
thereof is obtained as resolution of the input image. 

resolution = ( £ resol of blocks assigned 
conformity code) / number of blocks assigned 
5 conformity code 

The term "resolution 11 used in the description 
represents a read resolution in the case that an input 
image is applied to a prepared model rather than a 
read resolution of an input image that is read by a 

10 scanner. 

Thereafter, a color labeling process is performed 
(at step S13). In the color labeling process, 
adjacent pixels with similar colors are assigned the 
same label. Enclosing rectangles that are connected 

15 areas with the same labels as the label image are 
obtained. As information of enclosing rectangles, the 
coordinate values of the enclosing rectangles, the 
average color (R, G, B) of the connected areas of the 
enclosing rectangles, the label numbers of the 

20 connected areas of the enclosing rectangles, and the 
area of the connected areas (the number of pixels of 
the connected areas ) are stored . 

In reality, the input image is scanned from the 
upper left position. A non-labeled pixel is 

25 considered. With the RGB luminance value and the 
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estimated resolution of the pixel, the color 
difference table is searched for the maximum value of 
luminance values to adjacent pixels for each of RGB. 
The maximum value is treated as a threshold value for 
5 labeling a considered pixel. 

In the labeling process, for an area with the same 
label expanded from one pixel, whenever one pixel is 
added, the average color (R, G, B) in the label area 
is obtained. It is determined whether or not the same 

10 label as the label area is assigned to an adjacent 
pixel in the label area. When the color of the 
adjacent pixel to be newly labeled is different from 
the average color of the label area that has been 
labeled by the threshold value or more, even if the 

15 color difference of the adjacent pixel is not larger 
than the threshold the adjacent pixel to be newly 
labeled is assigned a label different from the label 
area that has been labeled. Thus, even if the color 
of the boundary of the character area and the 

20 background area gradually varies, the character area 
can be correctly extracted. 

Fig. 25 is a schematic diagram for explaining a 
color labeling process according to an embodiment of 
the present invention. 

25 In Fig. 25, it is assumed that the color of a 
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pixel e is black and that colors of pixels that are 
apart from the pixel e vary from black to red. When 
the color difference between the pixel e and adjacent 
pixels a to i except for e is the threshold value or 
5 less, the pixels e and the adjacent pixels a to i 
(other than e) are assigned the same label. 

When the pixel j is assigned a label, it is 
determined whether or not the color difference between 
the pixels f and j is the threshold value or less and 

10 whether or not the color difference between the 
average value of the colors of the pixels a to i and 
the color of the pixel j is the threshold value or 
less. Even if the color difference between the pixel 
f and the pixel j is the threshold value or less, when 

15 the color difference between the average value of the 
colors of the pixels a to i and the color of the pixel 
j exceeds the threshold value, the pixel j is assigned 
a label different from the label of the pixel f . 

When colors of pixels that are apart from the 

20 pixel e gradually vary from the color of the pixel e, 
although the color of the pixel j is similar to the 
color of the pixel f , the color difference between the 
pixels j and e is remarkable. Thus, when the average 
value of the colors of the pixels a to i that have 

25 been assigned a label is compared with the color of 



88 

the pixel j , the average value of the colors of the 
pixels a to i is affected by the color of the pixel 
e. Thus, this color difference is larger than the 
color difference between the pixels f and j . 
5 Thus, even if colors gradually vary, a boundary 

can be formed between the pixels f and j . 
Consequently, a black area and a red area can be 
prevented from being assigned the same label . 

Next, a real method for an R luminance value will 

10 be described. 

A resolution value of the table for R is fixed to 
an estimated resolution value. The table for R is 
scanned for a luminance value R of a considered pixel 
that satisfies the condition Ri < R < Ri + 1. At this 

15 point, assuming that the maximum value of luminance 
values of adjacent pixels against the R luminance 
value Ri is Rdi and that the maximum value of 
luminance values of the adjacent pixels against the 
R luminance value Ri + 1 is Rdi + 1, the luminance 

20 value R of the considered pixel is obtained by 
linearly interpolating the R luminance value Ri of the 
table for R and the R luminance value Ri + 1 of the 
table for R. This relation is applied to the maximum 
value of the luminance values of the adjacent pixels 

25 so as to obtain the estimated maximum value delta_r 
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of the luminance values of adjacent pixels. In other 
words, the maximum value delta_r is obtained as 
follows. 

rl = Ri - Ri + 1 

rm = R - Ri 

rn = Ri + 1 - R 

delta_r = Rri * rn / rl + Rri + 1 * rm / rl 
This relation applies to G and B. 

In such a manner, with a color value (R, G, B) of 
a considered pixel that has not been processed, the 
maximum value (delta_r, delta_g, delta_b) of luminance 
differences between the considered pixel and the 
adjacent pixels is obtained as a threshold value for 
labeling adjacent pixels. 

Assuming that eight pixels adjacent to a 
considered pixel that has not been processed are 
scanned, that a pixel that has been assigned a label 
is present adjacent to the considered pixel, and that 
the RGB luminance value (Rs, Gs, Bs) of the labeled 
pixel satisfies the following conditions: 

|R - Rs | <= delta_r 

and 

|G - Gs| < = delta_ g 
and 

|B - Bs | <= delta_b, 
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since the color of the considered pixel is similar to 
the color of the adjacent pixel, the label of the 
adjacent pixel is stored as a current label region. 
The considered pixel is assigned the current label 
5 region. In addition, as the average color (Rm, Gm, 
Bm) of the area assigned the current label region, a 
new average color of which the color of the considered 
pixel is added to the average color of the adjacent 
labels is obtained. 

10 Otherwise, the considered pixel is assigned a new 

label value of the maximum label value plus 1 as the 
current label. In this case, as the average value 
(Rm, Gm, Bm) of the area, the RGB value of the 
considered pixel is assigned. 

15 In any case, the coordinate value of the 

considered pixel is stored in a process coordinate 
array . 

Next, a process for extending areas with the same 
color around a considered pixel is performed. In this 

20 process, the top coordinate value is extracted from 
the process coordinate array. A pixel with the top 
coordinate value is treated as a considered pixel. 
An estimated threshold value (delta_jr, delta_g, 
delta_b) that is the difference between the RGB 

25 luminance value of the considered pixel and the 
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luminance value of an adjacent pixel is obtained. 

The threshold value (delta__rm, delta_gm, delta_bm) 
of the luminance difference of each RGB signal of the 
average color is obtained with the average color (Rm, 
5 Gm, Bm) in the area and the estimated resolution. 

Next, as shown in Fig. 12A, eight pixels adjacent 
to the considered pixel are scanned so as to obtain 
(Ri, Gi, Bi) values (where i = 1 to 8) of the pixels 
adjacent to the considered pixel. The luminance 
10 difference value between the (Rc, Gc, Be) value of the 
color of the considered pixel and the (Rm, Gm, Bm) 
value of the average color of the adjacent pixels is 
obtained with respect to the RGB value. When the 
luminance difference value satisfies the following 
15 conditions: 

|R± - Rc| <= delta_r 

and 

|G± - Gc| <= delta_g 
and 

20 |Bi - Be | <= delta_b 

| Ri - Rm | <= delta_rm 
and 

| Gi - Gin | <= delta_gm 
and 

25 |Bi - Bml <= delta bm 
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the adjacent pixels are assigned the same label as 

that of the considered pixel and the coordinate values 
of the adjacent pixels are stored in the process 
coordinate array. The considered pixel is deleted 
5 from the process coordinate array. 

Next, the maximum value and the minimum value of 
the process coordinates (x, y) are obtained and stored 
as an enclosing rectangle of the label area along with 
the average color of the label area as an attribute 
10 of the enclosing rectangle. 

This process is repeated until there is no pixel 
in the process coordinate array. 

As results of the labeling process, an enclosing 
rectangle that surrounds an area with the same label 
15 as that of a label image is obtained. 

Output information of the labeling process is as 
follows . 

- Label image: 4 bytes/pixel 
Enclosing rectangles 
20 Number of rectangles: maxcount 

Rectangle information: lbtbl 

Label number of label image included: 
label 

Upper left coordinates of rectangle: 

25 (xmin, ymin) 
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Lower right coordinates of rectangle: 

(xmax, ymax) 

Representative color in rectangle: 

(r, g, b) 

Relevant group number: string_label 

Length of contour: contour 
Next, a considered rectangle extracting process is 
performed (at step S14). In the considered rectangle 
extracting process, an enclosing rectangle in the 
range of a predetermined size is extracted as a 
considered rectangle from enclosing rectangles with 
all colors. 

In reality, an enclosing rectangle whose x length 
is 23 dots or more (at 100 dpi) and 1/2 or less of the 
width of the image and whose y length is 23 dots or 
more (at 100 dpi) and 1/2 or less of the height of the 
image is extracted as a considered rectangle. 

In the considered rectangle extracting process, 
the sizes of enclosing rectangles can be limited as 
a reference for grouping them. For example, when a 
title area is extracted, with reference to an 
enclosing rectangle corresponding to the size of 
characters of a title, enclosing rectangles can be 
grouped. Thus, enclosing rectangles such as noise and 
a background that are not used in a title can be 
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prevented from being referenced in the grouping 
process* Consequently, since an unnecessary process 
can be prevented, the process speed can be improved. 
Next, a process for extracting a search range 
5 around a considered pixel and rectangles in the search 
range is performed (at step S15). In this process, 
a search range with a predetermined size is assigned 
around a considered rectangle so as to extract 
rectangles in the search range. 

10 Fig. 26 is a schematic diagram for explaining a 

method for assigning a search range around a 
considered rectangle according to an embodiment of the 
present invention . 

In Fig. 26, enclosing rectangles with rectangle 

15 numbers 1 to 30 are extracted. It is assumed that an 
enclosing rectangle 151 with the rectangle number 1 
is a considered rectangle. In addition, it is assumed 
that a search range 152 is assigned around the 
enclosing rectangle 151. As objects to be grouped 

20 with respect to the enclosing rectangle 151, only the 
enclosing rectangles with the rectangle numbers 2 to 
24 in the search range are selected. In contrast, the 
enclosing rectangles with the rectangle numbers 25 to 
30 that are not in the search range 152 are not 

25 selected as objects to be grouped with respect to the 
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enclosing rectangle 151. 

In the case that: enclosing rectangles are grouped, 
when the search range of the enclosing rectangles is 
limited around a considered rectangle, only a 
5 character string that is present in a predetermined 
range can be extracted. For example, when a title 
area is extracted, it is not necessary to consider 
characters that are apart from the title. Thus, the 
title area can be effectively extracted. 

10 In reality, it is assumed that the longer side of 

the considered rectangle 151 is denoted by len, that 
the smaller side thereof is 35 dots at 100 dpi, and 
that the search range around the considered rectangle 
is denoted by range_size. 

15 To obtain the enclosing rectangles in the range 

range_size around the considered rectangle at high 
speed, the following method is used. 

Fig. 27 is a schematic diagram for explaining a 
method for extracting,, rectangles in a search range 

20 around a considered rectangle according to an 
embodiment of the present invention. 

In Fig. 27, assuming that enclosing rectangles 161 
to 1 64 with rectangle numbers 1 to 4 have been 
extracted, rectangle numbers and the number of 

25 rectangles are obtained and registered corresponding 
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to individual XY coordinate values. 

Since the enclosing rectangles 161 to 164 are not 
present at positions of X coordinate values 1 to 4, 
the number of rectangles n ■ 0 and rectangle number 
5 = null are recorded. Since the enclosing rectangle 
164 with the rectangle number 4 is present at the 
positions of the X coordinate values 5 to 1-3, the 
number of rectangles n = 1 and the rectangle number 
- 4 are registered. Since the enclosing rectangle 163 

10 with the rectangle number 3 and the enclosing 
rectangle 164 with the rectangle number 4 are present 
at the position of the X coordinate value 14, the 
number of rectangles n = 2 and the rectangle numbers 
= 3 and 4 are registered. Since the enclosing 

15 rectangle 164 with the rectangle number 4 is present 
at the positions of the X coordinate values 15 and 16, 
the number of rectangles n = 1 and the rectangle 
number = 4 are registered. The similar operation is 
performed at the positions of all the XY coordinate 

20 values. 

Next, assuming that the enclosing rectangle 163 
with the rectangle number 3 is a considered rectangle 
and that a search range 165 is assigned around the 
enclosing rectangle 163, the XY coordinate values 

25 included in the search range 165 are obtained. 
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Rectangle numbers at the positions of the individual 
XY coordinate values are obtained. Rectangle numbers 
that are present at both the positions of the XY 
coordinate values are those of enclosing rectangles 
5 in the search range 165. For example, the rectangle 
numbers of the enclosing rectangle at the positions 
of the X coordinate values in the search range 165 are 
2 to 4. The rectangle numbers of the enclosing 
rectangles at the positions of the Y coordinate values 

10 in the search range 165 are 1 to 4. Thus, the 
rectangle numbers at both the positions of the XY 
coordinate values are 2 to 4. Consequently, the 
enclosing rectangles 162 to 164 with the rectangle 
numbers 2 to 4 are obtained as those in the search 

15 range 165. 

In reality, the following steps a) to d) are 
performed. 

a) Rectangle numbers are projected to XY axes of 
the entire image. 

20 A set of the number of rectangles and rectangle 

number is registered for each pixel on the XY axes of 
the entire image. For example, assuming that a 
rectangle number i is present in a range (xl, x2) on 
the X axis, the number i is included in a set of 

25 rectangle numbers in the range from coordinate values 
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xl to x2 on the X axis. The same operation is 
performed for all rectangles on the XY coordinates. 

b) A range range_size is assigned around a 
considered rectangle . 
5 c) Rectangles at the potions of the coordinate 

values on the X axis in the range range_size are ORed 
so as to extract a set of rectangle numbers. 

The same operation is performed on the Y axis. 

d) The set of rectangle numbers on the X axis and 
10 the set of rectangle numbers on the Y axis are ANDed 
so as to obtain a set of rectangles around the 
considered rectangle . 

In the rectangle number storing method shown in 
Fig. 27, a storage capacity for storing all rectangle 
15 numbers of the entire image is required for each 
coordinate value. Thus, a large storage capacity is 
required . 

To obtain the number of rectangles and rectangle 
numbers at individual coordinate values at high speed 
20 without an increase of the storage capacity of the 
memory, the following method is used. 

Figs. 28A, 28B, and 28C are schematic diagrams for 
explaining a method for storing rectangle information 
in a search range, or partially disposed in the area 
25 around a considered rectangle according to an 
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embodiment of the present invention. 

In Fig. 28A, it is assumed that enclosing 
rectangles 171 and 172 with rectangle numbers 1 and 
2 have been extracted, respectively, that the 
5 enclosing rectangle 171 is present at the positions 
of the X coordinate values 155 to 165, and that the 
enclosing rectangle 172 are present at the positions 
of the X coordinate values 160 to 163. Thus, as shown 
in Fig. 28B, for the enclosing rectangle 171, the 
10 rectangle number 1 is registered at the positions of 
the X coordinate values 155 to 165. For the enclosing 
rectangle 172, the rectangle number 2 is registered 
at the positions of the X coordinate values 160 to 
163. 

15 Next, as shown in Fig. 28C, the rectangle numbers 

1 and 2 are arranged in the order thereof and in the 
ascending order of the X coordinate values 155 to 165. 
Thus, the number of rectangles having an equal 
coordinate value can be obtained. 

20 In reality, the following steps i) to v) are 

performed. 

i) The horizontal lengths of all rectangles are 
added. A data structure having rectangle numbers and 
coordinate information is prepared corresponding to 
25 the total horizontal length of the rectangles. 
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ii ) The coordinate values of rectangles and the 
rectangle numbers are assigned to the data structure. 

iii) The data of the data structure is sorted by 
coordinate values. 

5 iv) The number of rectangles is counted at each 

coordinate value. A storage area for the number of 
rectangles is prepared corresponding to each 
coordinate value. The sorted data structure is 
scanned from the beginning. When the coordinate value 

10 does not vary, the number of rectangles is counted. 

When the coordinate value varies, the number of 
rectangles is settled and stored. 

v) A rectangle number with the same coordinate 
value is stored in the sorted data structure in the 

15 prepared area. The sorted data structure is scanned 
from the beginning. When the coordinate value does 
not vary, a rectangle number is stored in the storage 
area. 

Next, a same color group extracting process is 
20 performed (at step S16). In the same color group 
extracting process, as with the labeling process, a 
threshold value of luminance difference values of RGB 
is obtained with the average color of a pattern in one 
considered rectangle. Rectangles whose RGB luminance 
25 difference values to the considered rectangle are not 
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larger than the threshold value are extracted as 
rectangles in the same group and stored in a group 
information area. Next, with an enclosing rectangle 
that is newly added to the group, the same process is 
5 performed until there is no added rectangle. 

In reality, the distance between value (Rc, Gc, 
Be) of one considered rectangle and value (Ri, Gi, Bi) 
of one enclosing rectangle i of a set of rectangles 
around the considered rectangle in the RGB space is 
10 defined as follows. 

dist - |Rc - Ri | + |Gc - Gi | + |Bc - Bi| 
In addition, it is assumed that the luminance 
difference of RGB obtained with the color value (Rc, 
Gc, Be) of a considered rectangle and a resolution is 
15 denoted by delta_r, delta_g, deltajb. When the value 
(Rc, Gc, Be) of the considered rectangle and the value 
(Ri, Gi, Bi) of one enclosing rectangle i of a set of 
enclosing rectangles around the considered rectangle 
satisfy the following conditions, it is determined 
20 that the enclosing rectangle around the considered 
rectangle is included in the group of the considered 
rectangle. The enclosing rectangle i around the 
considered rectangle is stored in the group of the 
considered rectangle. 
25 dist < TH RGB ( TH_RGB - 20) 
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or 

( | Rc - Ri | < delta_r 
and 

| Gc - Gi | < delta_g 
5 and 

| Be - Bi | < delta_b ) 

In addition, the rectangle number that is newly 
added to the group is stored to a newly added 
rectangle area. 

10 Next, the top rectangle number is extracted as a 

newly considered rectangle from the newly added 
rectangle area. Enclosing rectangles around the newly 
considered rectangle are obtained. By repeating the 
steps, enclosing rectangles included in the group are 

15 obtained until there is no rectangle added to the 
group . 

In the above-described process, the following data 
structure of group information to be extracted is 
obtained . 

20 Number of rectangles in group: num 

Rectangle number that composes group: child 
Number of rectangles after overlapped 
rectangles are combined: overlap_num 

Rectangle information after overlapped 

25 rectangles are combined: overlap 
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(Data type: rectangle information lbtbl 
type) 

Direction code of character string in 
group: orient 
5 Number of character strings in group: 

gyou_num 

Rectangle information of character string: 
gyou 

Next, a process for extracting the average 

10 thickness of a character line in a group is performed 
(at step S17). In this process, a contour tracing 
process is performed for a pattern of a label image 
corresponding to an enclosing rectangle in a group so 
as to obtain the length of the contour. In other 

15 words, in the rectangle range of one enclosing 
rectangle included in a group, the contour tracing 
process is performed on the label image corresponding 
to the label number of the enclosing rectangle. The 
extracted length of the contour is assigned to an 

20 attribute of the length of the contour of the 
rectangle. The area of the pattern is divided by the 
length of the contour so as to obtain the average 
thickness of a character line. The obtained average 
thickness is registered as an attribute of the 

25 rectangle to the rectangle information. 
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Figs. 29A and 29B are schematic diagrams for 
explaining a method for calculating the thickness of 
a character line according to an embodiment of the 
present invention . 
5 In Fig. 29A, it is assumed that a pattern "7" has 

been extracted. By counting the number of pixels 
included in the pattern "7", the area S of the 
pattern "7" is calculated. 

Next, as shown in Fig. 29B, the contour of the 

10 pattern "7" is extracted. The length L of the 
contour is calculated. The area S of the pattern "7" 
is divided by the length L of the contour so as to 
obtain the average thickness W of the pattern " 7 " . 
Next, a process for re-grouping enclosing 

15 rectangles corresponding to the thickness of a 
character line is performed (at step S18). In the 
process, the enclosing rectangles in the group are re- 
grouped corresponding to the average thickness of the 
character line so as to structure new group 

20 information. In other words, for the enclosing 
rectangles in the group, steps S14 to SI 6 shown in 
Fig. 23 are repeated. When the same color group 
extracting process is performed at step SI 6, an 
estimated thickness of a character line ( = area of 

25 pattern in rectangle / length of contour of pattern) 
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is used in addition "to color information used at: step 
S15. 

Fig. 30 is a schematic diagram for explaining a 
process for re-grouping enclosing rectangles 
5 corresponding to the thickness of a character line 
according to an embodiment of the present invention. 

In Fig. 30, it is assumed that a character string 
"tf)l— 7 I (group 1)" and a character string " ff)V— 2 
( group 2 ) " are described with the same color and 
10 closely disposed. Thus, it is assumed that the 
character string n Zf)]/—^fl" and the character string 
Zf 2" are grouped in the same group 181. 
The thickness of the characters of the character 
string "£f)]/—7l" and the thickness of the characters 
15 of the character string "^^^2" are calculated. 

Characters with different thickness are categorized 
as different groups. Thus, the character string n if 
)V—*7l n is categorized as the group 182. The 
character string "t/JV— 7^2 " is categorized as the 
20 group 183. 

In this process, the background and the character 
pattern that have been categorized as the same group 
can be categorized as different groups. In addition, 
character strings with different fonts or different 
25 character sizes can be categorized as different 
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groups . 

In reality, with an estimated thickness of a 
character line of a first enclosing rectangle (thickl) 
and an estimated thickness of a character line of a 
second enclosing rectangle ( thick2 ) , the rate of the 
thicknesses of the character lines (thick_rate) can 
be obtained as follow. 

thick_rate = max (thickl, thick2) 

/ min (thickl, thick2 ) 

As with step S15, when the following conditions 
are satisfied, two enclosing rectangles are recorded 
as the same group. 

thick_rate < TH_THICK_RATE 

and 

(dist < TH_RGB 
or 

( | Rc - Ri | < delta_r 
and 

| Gc - Gi | < delta_g 
and 

| Be - Bi| < delta_b)} 
where it is assumed that TH_THICK_RATE is 2.5. 

This process is performed for all enclosing 
rectangles in the original group so as to categorize 
the enclosing rectangles as new groups. 
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When the thickness of a character line is 
obtained, it is necessary to obtain the contour of the 
pattern. However, to trace the contour at high speed, 
the following process is performed. 
5 Figs. 31A, 31B, 31C, and 31D are schematic 

diagrams for explaining a method for assigning a 
search start point of pixels to trace a contour 
according to an embodiment of the present invention. 
In Fig. 31A, assuming that the contour of a 

10 pattern composed of pixels a to i is obtained, a label 
image is raster- scanned from the upper left position. 
Thus, the pixel a is obtained. After the pixel a is 
obtained, with a search start point of a pixel a f that 
is one of eight pixels adjacent to the pixel a and 

15 that has been searched two pixels before the pixel a, 
. the eight pixels adjacent to the pixel a are searched 
clockwise. When the pixel e is obtained, as shown in 
Fig. 31B, with a search start point of a pixel e 1 that 
is one of eight pixels ad j acent to the pixel e and 

20 that is the next pixel of the pixel a that has been 
searched as a contour just before the pixel e, the 
eight pixels adjacent to the pixel e are searched 
clockwise. When the pixel i is obtained, as shown in 
Fig. 31C, with a search start point of a pixel i 1 that 

25 is one of eight pixels, adjacent to the pixel i and 
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that is the next pixel of the pixel e that has been 
searched as a contour just before the pixel i, the 
eight pixels adjacent to the pixel i are searched 
clockwise. Likewise, as shown in Fig. 31D, pixel a 
5 -* pixel e -* pixel i -> pixel h pixel g -* pixel f -» 
pixel b -» pixel c are extracted as the contour of the 
pattern composed of the pixels a to i. 

In reality, the following steps a) to f) are 
performed. 

10 a) Rectangle size process completion flags are 

prepared. As initial values, all the rectangle size 
process completion flags are set to "0". 

b) The rectangle range is scanned from the upper 
left position with the process completion flags and 

15 label image so as to search a pixel i with process 
completion flag = 0 and label number = label number 
of current rectangle. The contour length is set to 
0. 

c) Eight pixels adjacent to the pixel i are 
20 searched from the pixel 0 clockwise as shown in Fig. 

32. When there is a pixel with process completion 
flag = 0 and label number = label number of current 
rectangle, a direction code DC of the pixel is 
extracted . 

25 Fig. 32 is a schematic diagram showing a direction 
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code viewed from a current pixel according to an 
embodiment of the present invention. 

In Fig. 32, the pixel C is a current pixel. The 
pixels 0 to 7 are pixels adjacent to the current pixel 
5 C. The direction code DC is viewed from the current 
pixel C. From a pixel i, eight pixels are searched 
in order. 

d) As the next pixel that composes the contour, 
a pixel in the direction of the direction code DC 

10 detected from the current pixel C, is searched. At 
this point, the contour length that has been 
calculated is incremented by 1 . The pixel to be 
searched next is assigned a new current pixel C 1 . A 
pixel with process completion flag = 0 and label 

15 number - label number of current rectangle is searched 
from eight pixels adjacent to the current pixel C 1 . 
At this point, the start direction code for searching 
eight pixels adjacent to the new current pixel C is 
DC that is the remainder from the former direction 

20 code (DC + 5) which is divided by 8. The adjacent 
pixels are searched clockwise in the direction DC. 

Thus, since a pixel is prevented from being 
redundantly searched, the contour tracing process can 
be performed at high speed. 

25 e) The contour tracing process is continued until 
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a pixel at the start coordinates is searched. The 
contour length at the point is extracted as the 
contour length of one contour of a pattern with a 
label value in the rectangle, 

f) Steps c) to e) are performed for another 
contour of the same pattern. All the contour lengths 
are added and finally extracted as the contour length 
of the pattern. 

Thereafter, a process for deleting a large 
rectangle from a group is performed (at step S19). 
In the process, to remove noise, the average size of 
rectangles in the group is obtained. Corresponding 
to the average size, excessively large rectangles are 
deleted . 

Figs. 33A and 33B are schematic diagrams for 
explaining a process for deleting a large rectangle 
from a group according to an embodiment of the present 
invention . 

In Fig. 33A, assuming that enclosing rectangles 
192 to 201 are categorized as a group 191, the sizes 
of the enclosing rectangles 192 to 201 in the group 
191 are obtained. Assuming that the size of the 
enclosing rectangle 192 is much larger than the sizes 
of the enclosing rectangles 193 to 201, as shown in 
Fig. 33B, the enclosing rectangle 192 is deleted from 
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the group 191 and thereby a group 202 composed of the 
enclosing rectangles 193 to 201 are generated. 

In reality, the following steps a) to d) are 
preformed . 

5 a) A histogram of the heights of the enclosing 

rectangles in the group is created. A rectangle 
height freq_height that represents the maximum 
frequency value is obtained in the following method. 

i) With a gradation window, a convolutional 
10 calculation is performed for the histogram (namely, 

a moving sum thereof is obtained). In reality, the 
gradation window is composed of for example five 
pixels of "11111" . 

ii) The minimum height value (in reality, 23 dots 
15 at 100 dpi) is obtained beforehand. A rectangle 

height f req_height that is larger than the minimum 
height value and that represents the maximum frequency 
is obtained. 

Figs. 34A and 34B are schematic diagrams for 
20 explaining a method for calculating the heights of 
rectangles that represent the maximum frequency value 
according to an embodiment of the present invention. 

In Fig. 34A, when the frequence of the heights of 
the enclosing rectangles 192 to 201 shown in Fig. 33 A 
25 is obtained, with a gradation window 205, as shown in 
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Fig. 34B, the moving sum of the frequency of the 
heights of rectangles obtained in Fig. 34A is 
calculated. After the moving sum of the frequency of 
the heights of rectangles is obtained, the height of 
5 rectangle freq_height that represents the maximum 
frequency is obtained from the heights of the 
enclosing rectangles 192 to 201. 

b) When the height of rectangle freq_height that 
represents the maximum frequency value is smaller than 

10 the minimum height value (in reality, 23 dots at 100 
dpi), the minimum height value is treated as the 
height of rectangle freq__height that represents the 
maximum frequency value. 

c) A threshold value th_large that determines a 
15 large rectangle is obtained as th__large = freq_height 

*10. 

d) Enclosing rectangles whose vertical and 
horizontal sizes are smaller than th_large are kept 
in the group. The remaining enclosing rectangles are 

20 deleted as noise from the group. 

Next, a process for combining overlapped 
rectangles in a group is performed (at step S20). In 
this process, a rectangle that surrounds enclosing 
rectangles that overlap is newly created and 

25 registered to group information. 
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In other words, enclosing rectangles that overlap 
(even if enclosing rectangles partly overlap) are 
combined. An enclosing rectangle that surrounds the 
combined enclosing rectangles is created. 
5 Figs. 35A and 35B are schematic diagrams for 

explaining a process for combining overlapped 
rectangles in a group according to an embodiment of 
the present invention. 

In Fig. 35A, an enclosing rectangle 211 is 

10 included in an enclosing rectangle 212. The enclosing 
rectangle 212 overlaps with an enclosing rectangle 
213. In this case, the enclosing rectangles 211 to 
213 are combined. Thus, in addition to the enclosing 
rectangles 211 to 213, an enclosing rectangle 214 is 

15 created. 

In this process, when one character is composed of 
a plurality of separated patterns such as "IS" 
(meaning a picture in Japanese) and "Hi" (meaning a 
checking station in Japanese), such patterns can be 

20 integrally treated and the successive process can be 
effectively performed. 

Next, a process for extracting a character string 
from a group is performed (at step S21). In this 
process, the direction of a character string in a 

25 group is estimated. The character string in the 
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direction is extracted. Information of the direction 
and coordinates of the extracted character string is 
stored for each group. 

Figs. 36A and 36B are schematic diagrams for 
5 explaining a process for extracting a character string 
from a group according to an embodiment of the present 
invention. 

In Fig. 36A, assuming that enclosing rectangles 1 
to 29 are categorized as a group 221, an enclosing 

10 rectangle adjacent to the enclosing rectangle 1 at the 
leftmost position is searched on the right. When the 
enclosing rectangle 2 adjacent to the enclosing 
rectangle 1 is obtained, the obtained enclosing 
rectangle is assigned the rectangle number 1 . The 

15 rectangle number 2 of the obtained enclosing rectangle 
is changed to the rectangle number 1. Likewise, an 
enclosing rectangle adjacent to the obtained enclosing 
rectangle with the rectangle number 1 is searched on 
the right. When an enclosing rectangle is obtained, 

20 the rectangle number of the left side enclosing 
rectangle is assigned to the right side enclosing 
rectangle . 

When there is no enclosing rectangle adjacent to 
the enclosing rectangle with the rectangle number on 
25 the right, an enclosing rectangle adjacent to an 
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enclosing rectangle with the rectangle number 15 is 
searched on the right. When an enclosing rectangle 
adjacent to the enclosing rectangle with the rectangle 
number 15 is obtained , the rectangle number 15 is 
5 assigned to the obtained enclosing rectangle. 

In such a process, as shown in Fig. 36B, the 
rectangle number 1 is assigned to the enclosing 
rectangle adjacent to the enclosing rectangle with the 
rectangle number 1 on the right side. Likewise, the 

10 rectangle number 15 is assigned to an enclosing 
rectangle adjacent to the enclosing rectangle with the 
rectangle number 15 on the right. After the rectangle 
numbers are transferred, an enclosing rectangle 222 
that surrounds the enclosing rectangles with the same 

15 rectangle number 1 assigned is created. Likewise, an 
enclosing rectangle 223 that surrounds the enclosing 
rectangles with the same rectangle number 15 assigned 
is created. 

In reality, the following steps a) to e) are 
20 performed. 

a) The direction of a character string is 
estimated corresponding to the coordinates of 
enclosing rectangles in the group. 

b) The adjacent relation of a rectangle that is 
25 the closest (upper, lower, left, or right position) 
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to an enclosing rectangle in outward and inward 
directions is obtained. 

c) When a character string is arranged in the 
horizontal direction, from a rectangle at the leftmost 

5 position, the start rectangle number is transferred 
to a rectangle on the right until there is no 
rectangle on the right. This process applies to the 
vertical direction . 

d) A rectangle adjacent to a rectangle at the 
10 leftmost position is assigned the rectangle number 

thereof. The rectangle number assigned is transferred 
to a rectangle adjacent thereto on the right until 
there is no rectangle relation. 

e) After the above -de scribed process is performed 
15 for all rectangles in the group, rectangles with the 

same rectangle number assigned are obtained. An 
enclosing rectangle that surrounds the rectangles with 
the same rectangle number assigned is extracted as a 
character string. , 

20 Next, a process for combining separated characters 

of a character string is performed (at step S22 ) . In 
this process, a process for combining vertically 
separated characters of a horizontal character string 
is applied to rectangles of the character string ( in 

25 the case of a vertical character string, a process for 
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combining horizontally separated characters is applied 
to rectangles of the character string ) . The combined 
rectangles are substituted with an enclosing 
rectangle. In this case, the rectangles of the 
5 character string are vertically combined. In other 
words, in the case of a horizontal character string, 
respective rectangles that surround an upper stroke 
and a lower stroke of a character (for example, " d" ) 
are combined and processed as one rectangle. 

10 Figs. 37A and 37B are schematic diagrams for 

explaining a process for combining separated 
characters of a character string according to an 
embodiment of the present invention. 

In Fig. 37A, enclosing rectangles 222 and 223 that 

15 surround enclosing rectangles horizontally adjacent 
in a group 221 are generated. In the enclosing 
rectangle 222, enclosing rectangles 224 and 225 that 
are vertically adjacent and enclosing rectangles 226 
and 226 that are vertically adjacent are present. In 

20 the enclosing rectangle 223, enclosing rectangles 228 
and 229 that are vertically adjacent and enclosing 
rectangles 230 and 231 that are vertically adjacent 
are present .' 

In this case, as shown in Fig. 37B, an enclosing 
25 rectangle 232 that surrounds the enclosing rectangles 
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222 and 223 is generated. The enclosing rectangles 
222 and 223 are combined. An enclosing rectangle 233 
that surrounds the enclosing rectangles 226 and 227 
is generated. The enclosing rectangles 226 and 227 
5 are combined. An enclosing rectangle 234 that 
surrounds the enclosing rectangles 228 and 229 is 
generated. The enclosing rectangles 228 and 229 are 
combined. An enclosing rectangle 235 that surrounds 
the enclosing rectangles 230 and 231 is generated. 

10 The enclosing rectangles 230 and 231 are combined. 

In reality, the following steps a) to c) are 
performed. In the following example, the case of a 
horizontal character string will be described. 
However, these steps apply to the case of a vertical 

15 character string. 

a) Rectangles in a character string are sorted in 
the ascending order of X coordinate values. 

b) The first rectangle is assigned a current 
rectangle. When the next rectangle overlaps with the 

20 current rectangle in the range of the X coordinate 
values of the current rectangle, these rectangles are 
combined. The range of the X coordinate values of the 
current rectangle becomes the range of the X 
coordinate values of the combined rectangles. This 

25 process is repeated until there is no rectangle to be 
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combined. When there is no rectangle to be combined, 
the combined rectangles are stored. 

c) After there is no rectangle to be combined, 
the next rectangle is assigned a new current 
5 rectangle. With the new current rectangle, step b) 
is repeated. 

Thereafter, a process for re-grouping rectangles 
corresponding to the sizes of characters of a 
character string is performed (at step S23 ) . In this 
10 process, rectangles are re-grouped corresponding to 
vertical sizes of characters of a character string 
information of the group. The results are stored in 
new group information. 

Thus, character strings with the same color and 
15 different character sizes can be categorized as 
different groups. In addition, noise and a character 
string can be categorized as different groups. 

Figs. 38A and 38B are schematic diagrams for 
explaining a process, for re-grouping rectangles 
20 corresponding to the sizes of characters of a 
character string according to an embodiment of the 
present invention . 

In Fig. 38A, assuming that enclosing rectangles 
222 and 223 that surround enclosing rectangles that 
25 are horizontally adjacent in a group 221 have been 
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generated, a height HI of the enclosing rectangle 222 
and a height H2 of the enclosing rectangle 223 are 
calculated. When the difference between the height 
HI of the enclosing rectangle 222 and the height H2 
5 of the enclosing rectangle 223 is larger than a 
predetermined threshold, the enclosing rectangles 222 
and 223 are categorized as different groups. 

In reality, when the difference of the heights of 
two rectangles of a horizontal character string (the 

10 difference of the widths of two rectangles of a 
vertical character string) is smaller than the 
predetermined threshold value and the rectangles of 
the two rectangles overlap with each other or the 
distance between the two rectangles is smaller than 

15 the predetermined threshold value, the two rectangles 
are categorized as the same group and assigned the 
same group number. This process is performed for all 
groups . 

Next, a new group is generated with character 
20 strings with the same group number. In this process, 
character strings with different character sizes 
categorized as the same group at step S18 shown in 
Fig. 23 can be categorized as different groups. 

Thereafter, a process for deleting a large 
25 rectangle from a group is performed ( at step S24 ) . 
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In this process, to remove noise, the average size of 
rectangles in a group is obtained. With reference to 
the average size, rectangles with very large sizes are 
deleted again. In other words, for a group that has 
5 been varied in the above-described process, the 
process at step S19 shown in Fig. 21 is performed 
again so as to remove the noise in the group. 

Next, a group combining process is performed (at 
step S25). In this process, groups that are 

10 horizontally or vertically adjacent and that have the 
same color are combined. In other words, in the case 
that the colors of two groups are similar, that they 
are adjacent, and that enclosing rectangles of the 
groups are in a row, the two enclosing rectangles are 

15 combined as one group. This process is performed so 
as to determine a likelihood of character strings of 
groups corresponding to the arrangement of rectangles 
as will be described later. 

Figs. 39A and 39B are schematic diagrams for 

20 explaining a process for combining groups according 
to an embodiment of the present invention. In Fig. 
39A, assuming that an enclosing rectangle 241 of a 
group 1 and an enclosing rectangle 242 of a group 2 
are horizontally arranged, a distance LI between the 

25 enclosing rectangles 241 and 242, a coordinate 
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difference El between the upper edge of the enclosing 
rectangle 241 and the upper edge of the enclosing 
rectangle 242, and a coordinate difference E2 between 
the lower edge of the enclosing rectangle 241 and the 
5 lower edge of the enclosing rectangle 241 are 
obtained. In the case that the distance LI between 
the enclosing rectangles 241 and 242, the coordinate 
difference El between the upper edge of the enclosing 
rectangle 241 and the upper edge of the enclosing 

10 rectangle 242, and the coordinate difference E2 
between the lower edge of the enclosing rectangle 241 
and the lower edge of the enclosing rectangle 242 are 
smaller than respective threshold values and that the 
difference between the representative color of the 

15 group 1 and the representative color of the group 2 
is not larger than a predetermined threshold value, 
the enclosing rectangles 241 and 242 are combined and 
an enclosing rectangle 243 that surrounds the 
enclosing rectangles 241 and 242 is generated. 

20 In reality, in the case that the coordinates of 

enclosing rectangles of two groups are close and the 
distance thereof is short and that it is supposed that 
the two enclosing rectangles can be combined as one 
character string, the enclosing rectangles are 

25 categorized as one group in the following conditions. 
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- The luminance difference values of the average 
RGB value of two groups is smaller than a 
predetermined threshold value. 

- When two groups are horizontally arranged, the 
horizontal distance thereof is smaller than a fixed 
threshold value (when two groups are vertically- 
arranged, the vertical distance thereof is smaller 
than a fixed threshold value). 

- When two groups are horizontally arranged, the 
coordinate difference between the upper edges of 
enclosing rectangles of the two groups is smaller than 
a fixed threshold value and the coordinate difference 
between the lower edges of the enclosing rectangles 
of the two groups is smaller than a fixed threshold 
value (when two groups are vertically arranged, the 
coordinate difference between the right edges of 
enclosing rectangles of the two groups is smaller than 
a fixed threshold value and the coordinate difference 
between the left edges of the enclosing rectangles of 
the two groups is smaller than a fixed threshold 
value ) . 

This process is repeated until there is no group 
to be combined. 

Next, a process for extracting a group that is a 
likelihood of characters corresponding to the 
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arrangement of rectangles is performed (at step S26). 
In this process, for rectangles of a group that 
satisfy a particular condition such a size or a shape, 
when the upper edge or lower edge of one rectangle is 
close to the upper edge or lower edge of an adjacent 
rectangle, it is determined that the rectangles are 
in a row. In this case, a row flag is set. A row 
rectangle ratio that represents the number of 
rectangles that are in a row in those of the group 
that satisfy a predetermined condition is obtained. 
When a group with a row rectangle ratio that exceeds 
a predetermined threshold value is extracted as a 
character string. 

Figs. 40A and 40B are schematic diagrams for 
explaining a process for extracting a group that is 
a likelihood of a character string corresponding to 
the arrangement of rectangles according to an 
embodiment of the present invention. 

In Fig. 40A, enclosing rectangles 252 to 256 and 
enclosing rectangles 258 to 269 are generated with an 
input image. The enclosing rectangles 252 to 256 are 
categorized as one group. Thus, an enclosing 
rectangle 251 that surrounds the enclosing rectangles 
252 to 256 is generated. Likewise, the enclosing 
rectangles 258 to 269 are categorized as another 
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group. Thus, an enclosing rectangle 257 that 
surrounds the enclosing rectangles 258 to 269 is 
generated . 

Next, the sizes and positions of enclosing 
5 rectangles of groups are determined. An enclosing 
rectangle horizontally arranged is assigned a 
horizontal arrangement flag 1. An enclosing rectangle 
vertically arranged is assigned a vertical arrangement 
flag 2. Since the enclosing rectangles 252 to 256 are 

10 randomly arranged in the enclosing rectangle 251, the 
enclosing rectangles 252 to 256 are assigned neither 
the horizontal arrangement flag 1 nor the vertical 
arrangement flag 2. On other hand, since the 
enclosing rectangles 258 to 269 are horizontally 

15 arranged in the enclosing rectangle 257, the enclosing 
rectangles 258 to 269 are assigned the horizontal 
arrangement flag 1 . 

After the flag assigning process for the enclosing 
rectangles 252 to 256 and the enclosing rectangles 258 

20 to 269 has been completed, the ratio of rectangles 
assigned flags in all rectangles is calculated in each 
group. A group with a high rectangle ratio is 
extracted as a character string. A group with a low 
rectangle ratio is removed as noise. 

25 Thus, as shown in Fig. 40B, since the enclosing 
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rectangles 252 to 256 have not been assigned flags, 
the group thereof is removed as noise. In contrast, 
since the enclosing rectangles 258 to 269 have been 
assigned the horizontal arrangement flag 1, the group 
thereof is extracted as a character string. 

In reality, the following steps a) to e) are 
performed . 

a) When two adjacent rectangles in a group 
satisfy a predetermined condition of a size or a shape 
and the coordinate difference between the upper edges 
of the two rectangles is smaller than a predetermined 
threshold value (height WID_RATE (= 0.2) times), the 
two rectangles are assigned the horizontal arrangement 
flag 1. 

When two rectangles are adjacent rectangles, they 
should be present in the range of the width of a 
reference rectangle times WID_RATE_DIST (= 0.5). 

b) When two adjacent rectangles in a group 
satisfy a predetermined condition of a size or a shape 
and the coordinate difference between the left edge 
of one rectangle and the right edge of the other 
rectangle is smaller than a predetermined threshold 
value ( the width times WID_RATE ) , the two rectangles 
are assigned the vertical arrangement flag 2 . 

When two rectangles are adjacent rectangles, they 
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should be present in the range of the height of a 
reference rectangle times WID_RATE_DIST . 

c) The number of rectangles assigned the 
horizontal arrangement flag and the number of 

5 rectangles assigned the vertical arrangement flag are 
compared in the group. The direction of a character 
string corresponds to the compared result* 

d ) The ratio of rectangles arranged in the group 
is obtained by "number of rectangles arranged / total 

10 number of rectangles that satisfy the predetermined 
condition such as size or shape in the group". 

As a condition of a rectangle in a group, the 
height that represents the most frequency value of 
rectangles in the group is obtained by the method 

15 shown in Fig. 34. The height x 3 is denoted by 
large_rect. The height of the rectangle is larger 
than the fixed size (23 dots at 100 dpi) and smaller 
than large_jrect. 

e) When the ratio of rectangles arranged in a 
20 group is larger than a predetermined threshold value 

NUMERATE (= 0.8), the group is kept as a group that 
is a likelihood of a character string. Otherwise, the 
group is removed as noise. 

A center point " • " and a dash "-" in a group are 
25 kept as characters. 
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Next, a noise group deleting process is performed 
(at step S27). In this process, a group that is 
composed of two rectangles and that is treated as 
noise corresponding to the shape or coordinates is 
5 extracted and deleted. In other words, a group that 
is composed of two rectangles and of which the 
distance between the upper edge and the lower edge (or 
the left edge and the right edge) of the two 
rectangles is small is deleted as noise unless it is 

10 determined that the two rectangles are in a row. 

Fig. 41 is a schematic diagram for explaining a 
process for deleting a noise group according to an 
embodiment of the present invention. 

In Fig. 41, it is assumed that two enclosing 

15 rectangles 272 and 273 are extracted as one group and 
an enclosing rectangle 271 that surrounds the two 
enclosing rectangles 272 and 273 is generated. In 
this case, the lower edge of the enclosing rectangle 
272 accords with the . lower edge of the enclosing 

20 rectangle 273. However, since the upper edge of the 
enclosing rectangle 272 is largely different from the 
upper edge of the enclosing rectangle 273, the 
enclosing rectangles 272 and 273 are treated as noise 
and the group thereof is deleted from groups that are 

25 likelihood of character strings. 
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In reality, the following process is performed. 

Assuming that the average height of two rectangles 
of a group is denoted by hei, that the average width 
thereof is denoted by wid, and that the ratio DELJRATE 
5 for assigning a threshold value is 0.1, a group that 
satisfies the following conditions is kept, A group 
that does not satisfies the following conditions is 
deleted as noise. 

( [coordinate at upper edge of first rectangle - 
10 coordinate at upper edge of second rectangle | < hei 
x DEL_RATE 

and 

| coordinate at lower edge of first rectangle - 
coordinate at lower edge of second rectangle | < hei 
15 x DELJRATE 

or 

( (coordinate at left edge of first rectangle - 
coordinate at left edge of second rectangle | < wid x 
DEL_RATE 
20 and 

[coordinate at right edge of first rectangle - 
coordinate at right edge of second rectangle | < wid 
x DELJRATE) 

Next, a process is performed for extracting 
25 patterns with the same color from a group (at step 
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S28). In this process, the representative color of 
a group is obtained as the average color of 
representative colors of rectangles in the group. A 
pattern that has not been extracted and that has the 
5 similar color to the original image is extracted as 
a binary image in the range of the group. The 
extracted pattern is assigned a label for a binary- 
image so as to obtain an enclosing rectangle of 
connected portions. When the size of the obtained 

10 enclosing rectangle is larger than a predetermined 
threshold value, the enclosing rectangle is added as 
a part of a character pattern to the group. 

For example, the average color (RGB) of patterns 
in a group is obtained. Pixels with the similar color 

15 of the average color are obtained in the range of the 
enclosing rectangle of the group so as to generate a 
binary image. A pattern portion is removed from the 
binary image. The remaining binary image is assigned 
the label for the binary image so as to obtain a new 

20 enclosing rectangle. When the size and shape of the 
new enclosing rectangle exceed predetermined threshold 
values > a pattern in the new enclosing rectangle is 
added to the group. 

In this process, a pattern that is lost in the 

25 above-described processes due to character width 
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information, color deviation, and so forth and that 
is a part of a character can be re-extracted. An 
example of the pattern is a sonant letter " of a 
character 

5 Figs. 42A and 42B are schematic diagrams for 

explaining a process for extracting patterns with the 
same colors from a group according to an embodiment 
of the present invention. 

In Fig. 42A, with the process from step Sll to S27 

10 shown in Fig. 23, it is assumed that a character 
string " 5 U-^Mt^^^XD^^JV" is extracted from an 
input image and that an enclosing rectangle 280 that 
surrounds the character string is generated. In this 
example, sonant letters "* " are removed from and 

15 "A". 

In this case, since the difference between the 
color of the sonant letter "* " and the color of 
exceeds the predetermined threshold value, the sonant 
letter "* " is categorized as a group different from 

20 a group of . 

In addition, since the difference between the 
color of the sonant letter " and the color of "A" 
exceeds the predetermined threshold value, the sonant 
letter "* " is categorized as a group different from 

25 a group of "A". 
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In the process at step S16, since colors of local 
portions of a character string to be extracted are 
compared, when the color of the character string is 
locally varied due to color shearing, a part of a 
5 character drops. 

Thus, the sonant letter " of n {f n and the sonant 
letter "* " of "A" are re-extracted. When these 
sonant letters are re-extracted, corresponding to the 
entire character string " % U—/\)V{t^<D 3 OOj&ft^x;!/" , 

10 a representative color thereof is obtained. A pattern 
with a similar color of the representative color is 
extracted from the label image. Thus, the influence 
of the variation of the color of a local portion of 
the character string is alleviated. Consequently, the 

15 sonant letters " # " of n {f n and "A" can be extracted. 

When the sonant letters "* " are re-extracted, the 
extracting range of the sonant letters "* " is limited 
to the enclosing rectangle 280. When the extracting 
range is limited, the sonant letters "* " of the 

20 characters to be re-extracted can be prevented from 
protruding from the range. Thus, since the range of 
the pattern to be processed can be narrowed, the 
process can be performed at high speed without a 
deterioration of the extracting accuracy. 

25 Thus, as shown in Fig. 42B, the sonant letters 
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"* " of and "A*" are correctly extracted and 

thereby the character string " ifu— /VV{t^<D 3 OMSj^ 
zr)]/ n can be extracted from the input image. 

As the re-extracted results, when the character 
string ^0-AVH^©3O©^tT^" and noise with the 
same color are in the range of the enclosing rectangle 
280, since the noise is also extracted, a small 
pattern that has been re-extracted is removed as 
noise. 

In reality, the following steps a) to h) are 
performed. 

a) Representative color information stored in 
rectangle information of a group is averaged for all 
rectangles in the group so as to obtain a 
representative color (Rm, Gm, Bm) of the group. 

b) Most frequency values of the widths and 
heights of rectangles of the group are obtained as mfw 
an mfh, respectively. When the width mfw and the 
height mfh of the most frequency values are larger 
than a threshold value minsize (= 2) of the size of 
the minimum rectangle, the width mfw and the height 
mfh of the maximum frequency values are substituted 
with minsize. 

c) An enclosing rectangle of the group is 
obtained. In the range of the enclosing rectangle, 
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a binary image is generated. All pixels of the binary 
image are set to "0". 

d) The colors of the pixels of the original image 
in the range of the enclosing rectangle of the group 

5 are checked. The position of a pixel of the original 
image is recorded when the color (R, G, B) of the 
pixel of the original image and the representative 
color of the group satisfy the following relation. 
Pixels at the same position of the binary image are 
10 set to "1". 

|Rm - R | + | Gm - G | + | Bm - B | < THR x 10 (THR = 

10) 

e) Rectangles of the group are extracted from the 
binary image. Patterns that have been extracted are 

15 removed. Only patterns that have been lost in the 
preceding processes and that have colors similar to 
the representative color of the group are kept in the 
binary image . 

f ) The binary image is assigned the label for the 
20 binary image so as to obtain an enclosing rectangle 

of connected areas. 

g) Rectangles that satisfy the following 
conditions are kept. The remaining rectangles are 
deleted. 

25 w < mfw / 2 and h < mfh / 2 
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where w is the width of an enclosing rectangle and h 
is the height thereof. 

h) A pattern of connected rectangles that compose 
the remaining rectangles is added to the group. 
5 Next, a process for determining an output order of 

groups is performed (at step S29). In the process, 
when a plurality of groups that contain character 
strings are extracted, the areas of enclosing 
rectangles of the groups are calculated. The groups 

10 are sorted in the order of the largest area. The 
sorted result becomes the output order. 

Thereafter, a process for generating a binary 
pattern is performed (at step S30). In the process, 
a pattern that composes a group is generated as a 

15 binary image so that the pattern can be recognized as 
characters . 

In reality, the following steps a) to c) are 
performed. 

a) A binary image with the size of an enclosing 
20 rectangle of a group is generated. All pixels of the 

binary image are set to " 0 " . 

b) A label number of a rectangle in the group is 
extracted. A label image is searched in a relevant 
rectangle. A pixel with the label number is 

25 extracted. The pixel at the coordinates in the binary 
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image is set to "1". 

c) The step b) is performed for all rectangles. 
A pattern that composes the group is generated as a 
binary image. 

5 The above-described process is described with an 

example of a real color image. 

Fig. 43A is a schematic diagram showing an example 
of an input original image. In the original image, 
headline characters 301 and 302, a picture 303, text 
10 characters 304, a frame 305, and so forth have been 
printed in colors. The headline characters 301 and 
302 are extracted as titles. Moreover, the headline 
characters 301 and 302 have been printed with one 
color. 

15 Fig. 43B is a schematic diagram showing the label 

image shown in Fig. 43 A. 

In Fig. 43B, the label image is obtained by the 

color labeling process at step S13 shown in Fig . 23 . 

In the color labeling process, connected areas with 
20 the same color are assigned the same label. An 

enclosing rectangle of patterns with the same label 

is generated. At this point, the number of enclosing 

rectangles amounts to around several thousands to 

several ten thousands . 
25 Fig. 43C is a schematic diagram showing considered 
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rectangles obtained from the label image shown in Fig. 
43B. 

In Fig. 43C, the considered rectangles are 
obtained by the considered rectangle extracting 
5 process at step S14 shown in Fig. 23. In the example 
shown in Fig. 43C, considered rectangles with 
rectangle numbers 1 to 64 are extracted. 

In the considered rectangle extracting process at 
step S14, among enclosing rectangles obtained from the 

10 label image , enclosing rectangles with a predetermined 
size are treated as considered rectangles. Thus, 
enclosing rectangles of the headline characters 301 
and 302 that should be extracted are dropped from 
areas 311 and 312 shown in Fig. 43C. In this example, 

15 the headline characters 302 that should be extracted 
in the area 312 shown in Fig. 43C is an enclosing 
rectangle corresponding to a character "IC" shown in 
Fig. 43 A. However, in the label image shown in Fig. 
43B, three strokes that compose the character "\Z" are 

20 assigned different labels. Thus, the sizes of the 
enclosing rectangles of the strokes become small. 

In the considered rectangle extracting process, 
when enclosing rectangles are categorized as groups, 
an enclosing rectangle 313 (not shown) corresponding 

25 to the text characters 304, an enclosing rectangle of 
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noise, and an enclosing rectangle of the frame 305 can 
be removed from a reference enclosing rectangle in the 
search range. Thus, the process can be performed at 
high speed. In the state shown in Fig. 43B, the 
5 number of enclosing rectangles amounts to several 
thousands to several ten thousands. On the other 
hand, in Fig. 43C, the number of enclosing rectangles 
decreases to several tens. 

Fig. 43D is a schematic diagram showing a group 

10 extracted result. 

In Fig. 43D, the group extracted result is 
obtained by the process for extracting groups with the 
same color at step S16 shown in Fig. 23. In Fig. 43D, 
enclosing rectangles with rectangle numbers 1 to 21 

15 are generated. In this process, enclosing rectangles 
that are obtained from the label image shown Fig. 43B, 
that are in a predetermined range of a considered 
rectangle shown in Fig. 43C and that have similar 
representative colors are categorized as the same 

20 group. An enclosing rectangle that surrounds the 
enclosing rectangles categorized as the same group is 
generated . 

Thus, an enclosing rectangle 321 of a group 
corresponding to the headline characters 301 " 0^ 
25 4 0 0ttgft7>^r-h" shown in Fig. 43A is generated. 
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In addition, an enclosing rectangle 322 of a group 
corresponding to the headline characters 302 
"KftWffOBHKltt^Ji" is generated. In the grouping 
process, the respective character strings that compose 
5 the headline characters 301 and 302 can be combined 
to one character string. In addition, enclosing 
rectangles 323 and 324 of the headline characters 301 
and 302 that should be extracted in the areas 311 and 
312 shown in Fig. 43C can be correctly extracted. 

10 When a label image of another enclosing rectangle 

with a similar representative color cannot be obtained 
in the predetermined range of a considered rectangle 
shown in Fig. 43C, since the considered rectangle does 
not compose a group, the considered rectangle is 

15 removed. 

Fig. 44A is a schematic diagram showing a re- 
grouped result of which the group extracted result 
shown in Fig. 43D is re-grouped corresponding to the 
thickness of a character line. 

20 In Fig. 44A, the re-grouped result is obtained by 

the process for re-grouping enclosing rectangles 
corresponding to the thickness of a character line as 
step S18 shown in Fig. 23. Enclosing rectangles with 
rectangle numbers 0 to 12 are generated as a group. 

25 In this process, an enclosing rectangle 331 of a group 
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corresponding to the headline characters 301 " 
4 0 0&|£##7> > 5 r — h" shown in Fig. 43A is generated. 
In addition, an enclosing rectangle 332 of a group 
corresponding to the headline character 302 "SIW 
5 ffi^MBlCS&^&M" shown in Fig. 43A is generated. 

Among enclosing rectangles shown in Fig. 43D, 
those that cannot be categorized as groups 
corresponding to the thickness of a character line are 
removed . 

10 Fig. 44D is a schematic diagram showing a result 

of which rectangles that overlap are combined. 

In Fig. 44D, the combined result of the rectangles 
is obtained by the process for combining overlapped 
rectangles of a group at step S20 shown in Fig. 23. 

15 Enclosing rectangles with rectangle numbers 0 to 12 
are generated. In this process, since most of 
enclosing rectangles in an area 333 corresponding to 
the picture 303 overlap, they are combined and 
removed. In the combing process, the number of 

20 enclosing rectangles to be processed is decreased. 
Thus, the process can be performed at high speed. 

Fig. 44C is a schematic diagram showing results of 
which groups that are likely to be character strings 
are extracted from an input image. 

25 In Fig. 44C, groups that are likely to be 
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character strings are obtained by the process for 
extracting groups that are likely to be character 
strings corresponding to the arrangement of rectangles 
at step S26 shown in Fig. 23. In Fig. 44C, enclosing 
rectangles with rectangle numbers 0 and 1 are 
extracted as a group. In this process, since 
enclosing rectangles of the group in an area 342 shown 
in Fig. 44B are randomly arranged, they are deleted. 
Thus, only an enclosing rectangle in an area 341 shown 
in Fig. 44B is extracted. 

Fig. 44D is a schematic diagram showing a result 
of which a pattern of a group is extracted from the 
image shown in Fig. 44C. 

In Fig. 44D, a pattern of a group is obtained by 
the process for extracting patterns with the same 
colors from a group at step S28 shown in Fig. 23. A 
pattern in an area of an enclosing rectangle of a 
group obtained in Fig. 44C is extracted. Thus, the 
headline characters 301 " B3£4 0 0 4tfc##7 h " and 
headline characters 302 "SIB W^<Z)M£»&H" shown 
in Fig. 43A are obtained. 

Fig. 45A is a schematic diagram showing a pattern 
of a first alternative of a title extracted from the 
image shown in Fig. 43A. Fig. 45B is a schematic 
diagram showing a pattern of a second alternative of 
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the title extracted from the image shown in Fig. 43A. 

In the example shown in Figs. 45A and 45B, 
corresponding to the sizes of areas of enclosing 
rectangles 401 and 402 of extracted patterns, the 
5 output order of titles extracted from the original 
image shown in Fig. 43A is assigned. 

Thus, the headline characters 302 "HB^F^SIW^S 
llft^^i" shown in Fig. 43A are presented as the first 
alternative of the title. The headline characters 301 

10 0 0ft^##7>^-b" shown in Fig. 43A are 

presented as the second alternative of the title. 

As described above, according to the present 
invention, the threshold value of the distance for 
determining whether or not to assign the same label 

15 to the adjacent pixels can be assigned corresponding 
to color information of the considered pixel, not a 
fixed value. Thus, it can be accurately determined 
whether or not the pixels are in the same range. 
Consequently, pixels can be accurately extracted from 

20 a desired area. 

Thus, according to an aspect of the present 
invention, areas regarded as the same color can be 
obtained with a reduced image. Consequently, a 
process for extracting areas regarded as the same 

25 color can be performed at high speed. 
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Thus, according to an aspect of the present 
invention, enclosing rectangles of label images and 
labels, representative colors thereof, the number of 
colors in the same label area, and colors are obtained 
5 from a reduced image. Corresponding to the obtained 
results, a rectangle of the original image is 
obtained. The labeling process can be performed in 
the rectangle. Thus, the labeling process can be 
performed at high speed. 

10 Thus, according to an aspect of the present 

invention, all colors and similarities are obtained 
from a reduced image, and similarities of areas of the 
reduced image are determined. When the similarity of 
colors of the areas is smaller than the predetermined 

15 threshold value, the same label can be assigned to the 
areas. Thus, a desired area can be extracted as 
viewed by the naked eye. 

Thus, respective threshold value elements are 
assigned three color elements. When all the three 

20 color elements of two colors are less than the 
respective threshold value, these colors can be 
treated as the same color. Consequently, the capacity 
of the table of the threshold values can be decreased. 
In addition, the labeling process can be very 

25 accurately performed. 
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According to an aspect of the present invention, 
patterns of an input image are categorized 
corresponding to color information and geometry 
information of the patterns thereof . Thus , when a 
5 headline area with the same color is extracted from 
a color input image, a part of the color input image 
is limited as the headline area. Patterns with the 
same colors are extracted. A same color pattern apart 
from the headline area is removed from an object to 

10 be categorized. In addition, a small pattern and a 
large pattern that are not headline characters can be 
removed from an object to be categorized corresponding 
to color information. Thus, the speed of the 
extracting process of the headline area can be 

15 improved. 

According to an aspect of the present invention, 
to cluster pixels of an input image, color information 
of a considered pixel and color information of a pixel 
adjacent thereto are cpmpared. When all pixels of the 

20 input image are compared, the number of times of the 
comparing process of color information of adjacent 
pixels becomes the square of the number of pixels of 
the input image. In contrast, when the color 
information of a considered pixel is compared with the 

25 color information of a pixel adjacent thereto, the 
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number of times of the comparing process becomes the 
number of pixels of the input image « Thus , the 
clustering process can be performed at high speed. 

In addition, when pixels of an input image are 
5 categorized as groups, clusters in a predetermined 
range are categorized as groups. Thus, pixels in the 
same cluster can be integrally handled. Thus, it is 
not necessary to process individual pixels. 
Consequently, the grouping process can be performed 

10 at high speed. In addition, in a limited range of the 
input area, clusters with similar colors can be 
categorized as the same group. Thus, the grouping 
process can be performed at further higher speed. 

According to an aspect of the present invention, 

15 an area in the range of a color deviation determined 
corresponding to a predetermined resolution is 
extracted as a same color area. Thus, since the color 
of a pattern of an input image is represented in a 
combination of sizes of dots of basic colors, even if 

20 an area represented as the same color in the input 
image is detected as a set of different colors 
depending on a read resolution, when the input image 
is read, the range of the same color of the area can 
be accurately extracted. Consequently, a part of a 

25 pattern represented as the same color in the input 
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image can be prevented from being dropped or lost. 

According to an aspect of the present invention, 
among enclosing rectangles generated with label 
images, an enclosing rectangle that becomes a 
5 reference for assigning a search area can be limited 
to an enclosing rectangle in a range of a 
predetermined size. Thus, when a character area is 
extracted from an input image, enclosing rectangles 
of a small pattern such as noise and a large pattern 

10 such as a background or a frame can be prevented from 
being selected as a reference of a search area for 
searching a character string. Consequently, a pattern 
that is not a character string can be suppressed from 
being categorized as a group. Thus, only a character 

15 string can be effectively extracted. 

Since a search area for searching a character 
string is assigned in a predetermined range from a 
considered enclosing rectangle, only characters in the 
vicinity of a considered enclosing rectangle can be 

20 searched. When a title or the like is extracted from 
the input image, a character that is apart from the 
title and that does not compose the title can be 
prevented from being extracted as the group of the 
title. Thus, only characters of the title can be 

25 effectively extracted. 
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According to an aspect of the present invention, 
in the case that a color gradually varies, only when 
color information of adjacent pixels is compared, 
since the colors of the adjacent pixels are similar, 
5 these pixels are assigned the same label . When these 
pixels are traced, the difference of colors cumulates. 
Thus, pixels with different colors may be assigned the 
same label. However, according to the present 
invention, since color information of a pattern that 

10 has been assigned a label can be compared, the 
labeling process can be performed corresponding to the 
cumulated value of the difference of colors. Thus, 
when an area whose color gradually varies is assigned 
a label corresponding to the compared result of the 

15 color information of the adjacent pixels, pixels with 
different colors can be prevented from being assigned 
the same label . 

According to an aspect of the present invention, 
a threshold value is assigned for a labeling process 

20 corresponding to the color difference between adjacent 
pixels in an area represented by a particular color. 
Thus, since an input image is printed by halftone 
printing method, even if the input image is 
represented in the same color, it may be detected as 

25 a ' set of different colors depending on a read 
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resolution. In this case, an area with the same color 
can be accurately extracted. Thus, a part of a 
pattern can be prevented from being dropped or lost 
from an area represented as the same color in the 
input image . 

According to an aspect of the present invention, 
a color generated by the halftone printing method is 
virtually generated on a computer. The variation of 
a color by the halftone printing method can be 
distinguished without need to analyze the variation 
of the color by the halftone printing method using a 
real printed document. 

According to an aspect of the present invention, 
a color difference table that stores luminance values 
of colors and the color difference between adjacent 
pixels corresponding to the luminance values in each 
read resolution of the print model is generated. 
Thus, regardless of a resolution of a device that 
reads data from a real, printed document, a threshold 
value for determining whether or not colors that are 
read from a real printed document are the same can be 
easily obtained. 

According to an aspect of the present invention, 
the read resolution of the print mode is estimated for 
an input image. Thus, a real printed document that 
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is read by a device can be handled with a print model 
virtually generated on a computer. The labeling 
process can be performed with a threshold value 
generated with a print model to the real printed 
5 document that is read by the device. 

According to an aspect of the present invention, 
a resolution of which a luminance value of a color and 
a color difference registered in the color difference 
table optimally confirms to the entire input image is 

10 estimated as a read resolution of the print model of 
the input image. Thus, for a real printed document 
that is read by a device, an optimum threshold value 
generated by a print model is applied to the entire 
input image so as to perform the labeling process. 

15 According to an aspect of the present invention, 

a resolution of which a luminance value of a color and 
a color difference registered in the color difference 
table optimally confirms to a local area of the input 
image is estimated as a read resolution of the print 

20 model of the local area of the input image. Thus, in 
an input image that is read by a device from a real 
printed document, for an area that exceeds a 
predetermined threshold value of the entire input 
image, the labeling process can be performed for the 

25 area with a threshold value optimum for the area. 
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Consequently, the accuracy of the labeling process can 
be improved . 

According to an aspect of the present invention, 
a read resolution of the print model is a fixed value. 
5 Thus, the model resolution estimating process of a 
real printed document can be omitted. Consequently, 
the process can be performed at high speed. 

According to an aspect of the present invention, 
label images can be categorized as a group. Thus, a 

10 character area extracted by the labeling process can 
be categorized as a character string area. 
Consequently, a title can be effectively extracted. 

According to an aspect of the present invention, 
a label image is categorized as the same group 

15 corresponding to color information and geometry 
information of an enclosing rectangle of the label 
image. Thus, when a headline area with the same color 
is extracted from an input color image, a pattern is 
extracted from a limited area of the input color 

20 image. A pattern that is apart from a headline area 
can be removed from an object to be categorized. In 
addition, a small pattern such as noise and a large 
pattern such as a background that is not the headline 
characters can be omitted from an object to be 

25 processed. Consequently, the speed of the extracting 
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process for the headline area can be improved. Since 
a labeled pattern is represented by an enclosing 
rectangle, a complicated pattern composing a character 
or the like can be represented in a simple shape. 
5 Thus, the storage capacity for the pattern can be 
decreased. In addition, when the relation of 
positions of patterns is obtained, the process can be 
simplified. 

According to an aspect of the present invention, 

10 representative colors of the patterns included in 
enclosing rectangles are compared and categorized as 
groups. Thus, patterns labeled can be compared as 
enclosing rectangles. When patterns are categorized 
as a group corresponding to color information, it is 

15 not necessary to compare each pixel that composes the 
patterns. Thus, since the number of elements to be 
compared can be decreased, the process can be 
performed at high speed. 

According to an aspect of the present invention, 

20 another enclosing rectangle present in a predetermined 
range of a considered enclosing rectangle is 
categorized as a group. Thus, when a title area or 
the like is extracted from an input image, the range 
of an enclosing rectangle to be categorized as a group 

25 can be limited to a range suitable for the title area. 
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The grouping process can be omitted for an area that 
protrudes from the title area. Consequently, the 
process can be preformed at high speed. 

According to an aspect of the present invention, 
5 rectangle numbers included in each of vertical and 
horizontal coordinates in a predetermined range of a 
considered enclosing rectangle are extracted. Thus, 
another enclosing rectangle in the predetermined range 
of the considered enclosing rectangle can be 

10 effectively searched. 

According to an aspect of the present invention, 
rectangle numbers of enclosing rectangles are stored 
in the order of coordinate values. Rectangle numbers 
are searched in the ascending order of the coordinate 

15 values. Rectangle numbers having an equal coordinate 
value are related to the coordinate value. Thus, when 
an area for storing a rectangle number of an enclosing 
rectangle is prepared for each coordinate value, the 
storage capacity corresponding to enclosing rectangles 

20 that are present in a predetermined range of a 
considered enclosing rectangle is required. In 
contrast, according to the present invention, since 
coordinate values corresponding to real enclosing 
rectangles are stored, the storage capacity can be 

25 decreased. 
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According to an aspect of the present invention, 
two enclosing rectangles are categorized as the same 
group when the two enclosing rectangles are present 
in a predetermined range of a considered enclosing 
5 rectangle and the color difference of the 
representative colors of the two enclosing rectangles 
is smaller than a color difference obtained from the 
color difference table. Thus, when a title area or 
the like is extracted from an input image, a range of 

10 enclosing rectangles to be categorized can be limited 
to a range suitable for the title area. When 
enclosing rectangles are compared and patterns are 
categorized as a group with color information, the 
variation of a color in the reading operation of a 

15 real printed document can be considered. Thus, the 
process can be preformed at high speed. In addition, 
the reading accuracy is improved. 

According to an aspect of the present invention, 
a label image is categorized as a group corresponding 

20 to the thickness of a pattern of the label image. 

Thus, since characters with different thicknesses can 
be categorized as different groups, when a title or 
the like is extracted from an input image, only a 
character string composed of characters with the same 

25 thickness can be extracted as a title. Thus, the 
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extracting accuracy of the title can be improved. 

According to an aspect of the present invention, 
the thickness of the pattern is obtained corresponding 
to the ratio of the area of the pattern and the 
contour length of the pattern. Thus, the thickness 
of the pattern can be accurately obtained. 

According to an aspect of the present invention, 
a pixel that has already been searched as the contour 
of the pattern is not searched. Thus, a pixel that 
becomes a contour of a pattern can be prevented from 
being re-extracted. Consequently, the speed of the 
contour tracing process can be improved. 

According to an aspect of the present invention, 
an enclosing rectangle that exceeds a predetermined 
range of the character size is deleted from the group 
of the enclosing rectangles. Thus, even if a pattern 
of a figure, a picture or the like is present in the 
vicinity of a title, and the pattern of the figure or 
picture is categorized, as a group of characters that 
composes the title, the pattern of the figure or 
picture can be removed from the group. Thus, the 
accuracy for extracting the title can be improved. 

According to an aspect of the present invention, 
a character string is extracted corresponding to the 
arrangement of enclosing rectangles. Thus, an area 
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of which enclosing rectangles are irregularly disposed 
and an area of which enclosing rectangles are 
regularly disposed can be distinguished. 
Consequently, an area of which enclosing rectangles 
5 are regularly disposed can be treated as a title area. 
Thus, only the title area can be effectively 
extracted . 

According to an aspect of the present invention, 
characters with different sizes are categorized as 

10 different groups. Consequently, when a title or the 
like is extracted from an input image, only a 
character string composed of characters with the same 
size can be treated as a title. Thus, the extracting 
accuracy for a title can be improved. 

15 According to an aspect of the present invention, 

it can be determined whether or not enclosing 
rectangles in a group are arranged in a row. A group 
of enclosing rectangles arranged in a row can be 
treated as a title area. Thus, a title can be 

20 effectively extracted from an input image. 

According to an aspect of the present invention, 
the representative color of a group is calculated 
corresponding to the representative color of patterns 
of a group. Thus, corresponding to all colors of a 

25 title area, the color of a title can be calculated. 
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The influence of color difference in a local area of 
the title area can be alleviated. Consequently, the 
extracting accuracy for the title area can be 
improved . 

According to an aspect of the present invention, 
adjacent enclosing rectangles of a group can be 
combined corresponding to the representative color of 
the group. Thus, even if characters composing the 
same title are categorized as different groups due to 
a slight color difference, these characters can be 
categorized as the same group. Consequently, a title 
area can be effectively extracted. 

According to an aspect of the present invention, 
adjacent enclosing rectangles of a group are combined 
when the difference of the sizes of the enclosing 
rectangles is in a predetermined range and the 
difference of representative colors of the enclosing 
rectangles in the group is in a predetermined range. 
Thus, when enclosing rectangles of a group are 
combined, only characters that compose the same title 
can be effectively combined. 

According to an aspect of the present invention, 
a pixel of an original image is re-extracted in the 
range of enclosing rectangles of the group 
corresponding to the representative color of the 
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group. Thus, even if characters composing a title 
include a sonant letter or a semi -sonant letter and 
the sonant letter or the semi-sonant is dropped from 
the group of the title due to a local color difference 
5 of the title area, the color of the sonant letter or 
the semi-sonant letter can be compared with the entire 
color of the title area so as to re-extract the sonant 
letter or the semi-sonant letter. Consequently, the 
extracting accuracy for the title can be improved. 

10 According to an aspect of the present invention, 

an enclosing rectangle of a pattern that has been re- 
extracted and that is larger than a predetermined 
threshold value is added to the group. Thus, when a 
sonant letter or a semi-sonant letter is re-extracted 

15 from the title area, even if a thin noise with the 
same color as the title is present in the title area, 
only the noise can be deleted. Consequently, the 
extracting accuracy for the title can be improved. 

According to an aspect of the present invention, 

20 the output order of patterns in a group is determined 
corresponding to the areas of enclosing rectangles 
categorized as the group, the positions thereof, and 
the relation of the positions thereof. Thus, even if 
a plurality of title alternatives are extracted from 

25 an input image, the likelihood of a title can be 
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evaluated. Consequently, title alternatives can be 
output in the order of the maximum likelihood of the 
title. 

According to an aspect of the present invention, 
a binary pattern is generated with a label image in 
enclosing rectangles categorized as a group. Thus, 
a pattern extracted from a color image can be 
effectively recognized. 

Although the present invention has been shown and 
described with respect to a best mode embodiment 
thereof, it should be understood by those skilled in 
the art that the foregoing and various other changes, 
omissions, and additions in the form and detail 
thereof may be made therein without departing from the 
spirit and scope of the present invention. 



